[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:53:11 dist-test-slave-vpvm krb5kdc[29134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:53:11 dist-test-slave-vpvm krb5kdc[29134](info): set up 2 sockets
Aug 09 07:53:11 dist-test-slave-vpvm krb5kdc[29134](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260809 07:53:12.285373 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:53:12 dist-test-slave-vpvm krb5kdc[29134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:53:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:53:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:53:12.400494 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:35757
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:35757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:12.510201 29151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:12.510473 29151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:12.510514 29151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:12.515367 29151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:12.515440 29151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:12.515462 29151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:12.515481 29151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:12.515506 29151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:53:12.520716 29151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:35757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:35757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:12.521718 29151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:12.522794 29151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:12.530951 29166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:12.531061 29151 server_base.cc:1034] running on GCE node
W20260809 07:53:12.530958 29162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:12.530978 29159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:12.531771 29151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:12.532810 29151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:12.534006 29151 hybrid_clock.cc:648] HybridClock initialized: now 1786261992534012 us; error 77 us; skew 500 ppm
Aug 09 07:53:12 dist-test-slave-vpvm krb5kdc[29134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261992, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:12.536695 29151 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:53:12.537567 29151 webserver.cc:460] Webserver started at http://127.28.112.62:40853/ using document root <none> and password file <none>
I20260809 07:53:12.538133 29151 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:12.538201 29151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:12.538421 29151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:12.540441 29151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "e2013ff978b54ba59f57f4c95f282655"
format_stamp: "Formatted at 2026-08-09 07:53:12 on dist-test-slave-vpvm"
server_key: "1ce5ae08ab3c00cc06b352790df885cb"
server_key_iv: "dd6646b5c8dbb767be3afcac5b616081"
server_key_version: "encryptionkey@0"
I20260809 07:53:12.540995 29151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "e2013ff978b54ba59f57f4c95f282655"
format_stamp: "Formatted at 2026-08-09 07:53:12 on dist-test-slave-vpvm"
server_key: "1ce5ae08ab3c00cc06b352790df885cb"
server_key_iv: "dd6646b5c8dbb767be3afcac5b616081"
server_key_version: "encryptionkey@0"
I20260809 07:53:12.544643 29151 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:53:12.547113 29182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:12.548317 29151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:12.548439 29151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "e2013ff978b54ba59f57f4c95f282655"
format_stamp: "Formatted at 2026-08-09 07:53:12 on dist-test-slave-vpvm"
server_key: "1ce5ae08ab3c00cc06b352790df885cb"
server_key_iv: "dd6646b5c8dbb767be3afcac5b616081"
server_key_version: "encryptionkey@0"
I20260809 07:53:12.548555 29151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:12.571276 29151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:12.573864 29151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:12.574051 29151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:12.580601 29151 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:35757
I20260809 07:53:12.580607 29263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:35757 every 8 connection(s)
I20260809 07:53:12.581490 29151 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:53:12.584215 29265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:12.586275 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 29151
I20260809 07:53:12.586434 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:53:12.588035 29120 external_mini_cluster.cc:1428] Setting key 36cf842281162ae62c99785327d2afe1
I20260809 07:53:12.590982 29265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655: Bootstrap starting.
I20260809 07:53:12.593197 29265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:12.593875 29265 log.cc:824] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:12.595710 29265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655: No bootstrap required, opened a new log
I20260809 07:53:12.599565 29265 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2013ff978b54ba59f57f4c95f282655" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 35757 } }
I20260809 07:53:12.599822 29265 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:12.599880 29265 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2013ff978b54ba59f57f4c95f282655, State: Initialized, Role: FOLLOWER
I20260809 07:53:12.600307 29265 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2013ff978b54ba59f57f4c95f282655" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 35757 } }
I20260809 07:53:12.600457 29265 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:53:12.600531 29265 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:53:12.600622 29265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:12.601398 29265 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2013ff978b54ba59f57f4c95f282655" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 35757 } }
I20260809 07:53:12.601716 29265 leader_election.cc:302] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2013ff978b54ba59f57f4c95f282655; no voters:
I20260809 07:53:12.602005 29265 leader_election.cc:288] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:53:12.602080 29270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:12.602248 29270 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655 [term 1 LEADER]: Becoming Leader. State: Replica: e2013ff978b54ba59f57f4c95f282655, State: Running, Role: LEADER
I20260809 07:53:12.602542 29270 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2013ff978b54ba59f57f4c95f282655" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 35757 } }
I20260809 07:53:12.603228 29265 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:12.603689 29270 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2013ff978b54ba59f57f4c95f282655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2013ff978b54ba59f57f4c95f282655" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 35757 } } }
I20260809 07:53:12.603806 29270 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:12.604004 29270 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2013ff978b54ba59f57f4c95f282655. Latest consensus state: current_term: 1 leader_uuid: "e2013ff978b54ba59f57f4c95f282655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2013ff978b54ba59f57f4c95f282655" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 35757 } } }
I20260809 07:53:12.604096 29270 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:12.604588 29276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 09 07:53:12 dist-test-slave-vpvm krb5kdc[29134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:12.607811 29276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:12.612622 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:12.591661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.28.112.62:35757)
0809 07:53:12.592118 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:12.592146 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:12.593108 (+ 962us) server_negotiation.cc:407] Connection header received
0809 07:53:12.596365 (+ 3257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:12.596387 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:12.596746 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:12.597082 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:12.598973 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:12.600393 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:12.601588 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:12.602158 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:12.602322 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:12.607001 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:12.607008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:12.607023 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:12.607056 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:12.608934 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:12.609495 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:12.609501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:12.609506 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:12.609622 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:12.610008 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:12.610012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:12.610013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:12.610541 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:12.610754 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:12.611397 (+ 643us) server_negotiation.cc:299] Negotiation successful
0809 07:53:12.611695 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":128,"threads_started":1}
I20260809 07:53:12.614459 29276 catalog_manager.cc:1347] Generated new cluster ID: 28116f28f5d0464da7660f4a67301d55
I20260809 07:53:12.614523 29276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:12.633191 29276 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:53:12.634424 29276 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:12.642459 29276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2013ff978b54ba59f57f4c95f282655: Generated new TSK 0
I20260809 07:53:12.643191 29276 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
I20260809 07:53:12.677170 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:35757
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:12.786321 29300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:12.786563 29300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:12.786623 29300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:12.790479 29300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:12.790529 29300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:12.790606 29300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:53:12.795568 29300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:35757
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:12.796561 29300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:12.797535 29300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:12.806706 29305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:12.806756 29300 server_base.cc:1034] running on GCE node
W20260809 07:53:12.806712 29307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:12.806706 29312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:12.807367 29300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:12.807920 29300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:12.809090 29300 hybrid_clock.cc:648] HybridClock initialized: now 1786261992809056 us; error 43 us; skew 500 ppm
Aug 09 07:53:12 dist-test-slave-vpvm krb5kdc[29134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261992, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:12.811779 29300 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:53:12.812598 29300 webserver.cc:460] Webserver started at http://127.28.112.1:39321/ using document root <none> and password file <none>
I20260809 07:53:12.813107 29300 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:12.813171 29300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:12.813351 29300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:12.815337 29300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "23643836c4ca47b481650923f51e7f30"
format_stamp: "Formatted at 2026-08-09 07:53:12 on dist-test-slave-vpvm"
server_key: "03b107f7274ff03d90342975fc5de29b"
server_key_iv: "a6e914ec0c9a12c482559340f1213fbd"
server_key_version: "encryptionkey@0"
I20260809 07:53:12.815866 29300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "23643836c4ca47b481650923f51e7f30"
format_stamp: "Formatted at 2026-08-09 07:53:12 on dist-test-slave-vpvm"
server_key: "03b107f7274ff03d90342975fc5de29b"
server_key_iv: "a6e914ec0c9a12c482559340f1213fbd"
server_key_version: "encryptionkey@0"
I20260809 07:53:12.819247 29300 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:53:12.821401 29324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:12.822512 29300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:12.822623 29300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "23643836c4ca47b481650923f51e7f30"
format_stamp: "Formatted at 2026-08-09 07:53:12 on dist-test-slave-vpvm"
server_key: "03b107f7274ff03d90342975fc5de29b"
server_key_iv: "a6e914ec0c9a12c482559340f1213fbd"
server_key_version: "encryptionkey@0"
I20260809 07:53:12.822710 29300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:12.837108 29300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:12.839985 29300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:12.840152 29300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:12.840691 29300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:12.841457 29300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:12.841502 29300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:12.841562 29300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:12.841588 29300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:12.850263 29300 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:34141
I20260809 07:53:12.850292 29487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:34141 every 8 connection(s)
I20260809 07:53:12.851068 29300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:53:12.852823 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 29300
I20260809 07:53:12.852958 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:53:12.853206 29120 external_mini_cluster.cc:1428] Setting key 299b2ddd0d65da17ba1e035fd677c8b1
Aug 09 07:53:12 dist-test-slave-vpvm krb5kdc[29134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261992, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.112.2@KRBTEST.COM; defaulting to no policy
I20260809 07:53:12.864121 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:12.852808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:44423 (local address 127.28.112.62:35757)
0809 07:53:12.852955 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:12.852960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:12.853790 (+ 830us) server_negotiation.cc:407] Connection header received
0809 07:53:12.855082 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:12.855085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:12.855133 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:12.855206 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:12.856309 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:12.856829 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:12.858177 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:12.858411 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:12.858498 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:12.860992 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:12.860996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:12.860998 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:12.861021 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:12.862303 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:12.862852 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:12.862855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:12.862857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:12.862934 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:12.863233 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:12.863235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:12.863236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:12.863397 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:12.863467 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:12.863895 (+ 428us) server_negotiation.cc:299] Negotiation successful
0809 07:53:12.864000 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 07:53:12.864817 29490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:12.853196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:35757 (local address 127.28.112.1:44423)
0809 07:53:12.853648 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:12.853697 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:12.854882 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:12.855352 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:12.855359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:12.855678 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:12.856156 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:12.856177 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:12.856941 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:12.856945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:12.858064 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:12.858074 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:12.858544 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:12.858549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:12.858717 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:12.859191 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:12.859206 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:12.860822 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:12.862415 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:12.862421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:12.862435 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:12.862755 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:12.863022 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:12.863024 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:12.863026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:12.863165 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:12.863467 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:12.863472 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:12.863694 (+ 222us) client_negotiation.cc:769] Sending connection context
0809 07:53:12.863845 (+ 151us) client_negotiation.cc:240] Negotiation successful
0809 07:53:12.864090 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.2@KRBTEST.COM" created.
I20260809 07:53:12.865996 29488 heartbeater.cc:347] Connected to a master server at 127.28.112.62:35757
I20260809 07:53:12.866205 29488 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:12.866648 29488 heartbeater.cc:511] Master 127.28.112.62:35757 requested a full tablet report, sending...
I20260809 07:53:12.868048 29209 ts_manager.cc:194] Registered new tserver with Master: 23643836c4ca47b481650923f51e7f30 (127.28.112.1:34141)
I20260809 07:53:12.869535 29209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:44423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
I20260809 07:53:12.880328 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:35757
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:12.988565 29493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:12.988934 29493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:12.988998 29493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:12.993816 29493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:12.993902 29493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:12.994009 29493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:53:12.999445 29493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:35757
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:13.000469 29493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:13.001506 29493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:13.010953 29501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:13.010962 29503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:13.010983 29512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:13.011317 29493 server_base.cc:1034] running on GCE node
I20260809 07:53:13.011734 29493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:13.012332 29493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:13.013518 29493 hybrid_clock.cc:648] HybridClock initialized: now 1786261993013499 us; error 27 us; skew 500 ppm
Aug 09 07:53:13 dist-test-slave-vpvm krb5kdc[29134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261993, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:13.016175 29493 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:53:13.017006 29493 webserver.cc:460] Webserver started at http://127.28.112.2:36197/ using document root <none> and password file <none>
I20260809 07:53:13.017531 29493 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:13.017596 29493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:13.017786 29493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:13.019767 29493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "0ea4391017184359becb2386763c2062"
format_stamp: "Formatted at 2026-08-09 07:53:13 on dist-test-slave-vpvm"
server_key: "726db314861a51005e955dfa47e9365a"
server_key_iv: "7aefae6a09e9330c5bdafc2597b9b7c5"
server_key_version: "encryptionkey@0"
I20260809 07:53:13.020301 29493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "0ea4391017184359becb2386763c2062"
format_stamp: "Formatted at 2026-08-09 07:53:13 on dist-test-slave-vpvm"
server_key: "726db314861a51005e955dfa47e9365a"
server_key_iv: "7aefae6a09e9330c5bdafc2597b9b7c5"
server_key_version: "encryptionkey@0"
I20260809 07:53:13.023809 29493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 07:53:13.026031 29523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:13.027397 29493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:13.027511 29493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "0ea4391017184359becb2386763c2062"
format_stamp: "Formatted at 2026-08-09 07:53:13 on dist-test-slave-vpvm"
server_key: "726db314861a51005e955dfa47e9365a"
server_key_iv: "7aefae6a09e9330c5bdafc2597b9b7c5"
server_key_version: "encryptionkey@0"
I20260809 07:53:13.027614 29493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:13.049407 29493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:13.051954 29493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:13.052109 29493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:13.052634 29493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:13.053431 29493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:13.053478 29493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:13.053519 29493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:13.053547 29493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:13.061707 29493 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:36019
I20260809 07:53:13.061728 29684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:36019 every 8 connection(s)
I20260809 07:53:13.062543 29493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:53:13.066193 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 29493
I20260809 07:53:13.066299 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:53:13.066573 29120 external_mini_cluster.cc:1428] Setting key 5847993eac307b2a74bf77d06dc31c70
Aug 09 07:53:13 dist-test-slave-vpvm krb5kdc[29134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261993, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:13.075783 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:13.064113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:34407 (local address 127.28.112.62:35757)
0809 07:53:13.064261 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:13.064266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:13.065044 (+ 778us) server_negotiation.cc:407] Connection header received
0809 07:53:13.066289 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:13.066293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:13.066349 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:13.066434 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:13.067674 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:13.068281 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:13.069374 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:13.069687 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:13.069785 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:13.072294 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:13.072297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:13.072299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:13.072322 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:13.073639 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:13.074276 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:13.074278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:13.074280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:13.074353 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:13.074699 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:13.074701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:13.074702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:13.074860 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:13.074947 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:13.075507 (+ 560us) server_negotiation.cc:299] Negotiation successful
0809 07:53:13.075637 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.28.112.3@KRBTEST.COM; defaulting to no policy
I20260809 07:53:13.076742 29691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:13.064471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:35757 (local address 127.28.112.2:34407)
0809 07:53:13.064904 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:13.064944 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:13.066089 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:13.066579 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:13.066585 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:13.066959 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:13.067541 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:13.067559 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:13.068405 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:13.068409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:13.069255 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:13.069265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:13.069859 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:13.069864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:13.070081 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:13.070596 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:13.070616 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:13.072145 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:13.073760 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:13.073767 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:13.073781 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:13.074186 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:13.074433 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:13.074437 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:13.074439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:13.074613 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:13.074958 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:13.074964 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:13.075261 (+ 297us) client_negotiation.cc:769] Sending connection context
0809 07:53:13.075453 (+ 192us) client_negotiation.cc:240] Negotiation successful
0809 07:53:13.075783 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":123,"threads_started":1}
I20260809 07:53:13.077745 29686 heartbeater.cc:347] Connected to a master server at 127.28.112.62:35757
I20260809 07:53:13.077977 29686 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:13.078428 29686 heartbeater.cc:511] Master 127.28.112.62:35757 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.3@KRBTEST.COM" created.
I20260809 07:53:13.079468 29209 ts_manager.cc:194] Registered new tserver with Master: 0ea4391017184359becb2386763c2062 (127.28.112.2:36019)
I20260809 07:53:13.079999 29209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:34407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
I20260809 07:53:13.093207 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:35757
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:13.201892 29695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:13.202133 29695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:13.202175 29695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:13.206246 29695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:13.206293 29695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:13.206363 29695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:53:13.211416 29695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:35757
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:13.212299 29695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:13.213219 29695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:13.222220 29702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:13.222297 29695 server_base.cc:1034] running on GCE node
W20260809 07:53:13.222256 29705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:13.222227 29701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:13.222808 29695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:13.223421 29695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:13.224572 29695 hybrid_clock.cc:648] HybridClock initialized: now 1786261993224559 us; error 24 us; skew 500 ppm
Aug 09 07:53:13 dist-test-slave-vpvm krb5kdc[29134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261993, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:13.227275 29695 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:53:13.228075 29695 webserver.cc:460] Webserver started at http://127.28.112.3:36743/ using document root <none> and password file <none>
I20260809 07:53:13.228588 29695 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:13.228662 29695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:13.228864 29695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:13.230845 29695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "1185fd2a0fed46b4b9ff1be92271cc6b"
format_stamp: "Formatted at 2026-08-09 07:53:13 on dist-test-slave-vpvm"
server_key: "94376f391b72d539bccd7cf695b50f92"
server_key_iv: "4aed02201f3821a9568d2f0ef5ad486c"
server_key_version: "encryptionkey@0"
I20260809 07:53:13.231411 29695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "1185fd2a0fed46b4b9ff1be92271cc6b"
format_stamp: "Formatted at 2026-08-09 07:53:13 on dist-test-slave-vpvm"
server_key: "94376f391b72d539bccd7cf695b50f92"
server_key_iv: "4aed02201f3821a9568d2f0ef5ad486c"
server_key_version: "encryptionkey@0"
I20260809 07:53:13.234920 29695 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 07:53:13.237164 29715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:13.238343 29695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 07:53:13.238448 29695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "1185fd2a0fed46b4b9ff1be92271cc6b"
format_stamp: "Formatted at 2026-08-09 07:53:13 on dist-test-slave-vpvm"
server_key: "94376f391b72d539bccd7cf695b50f92"
server_key_iv: "4aed02201f3821a9568d2f0ef5ad486c"
server_key_version: "encryptionkey@0"
I20260809 07:53:13.238536 29695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:13.256556 29695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:13.259133 29695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:13.259302 29695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:13.259855 29695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:13.260641 29695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:13.260690 29695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:13.260743 29695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:13.260771 29695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:13.269048 29695 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:42921
I20260809 07:53:13.269085 29887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:42921 every 8 connection(s)
I20260809 07:53:13.269876 29695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
Aug 09 07:53:13 dist-test-slave-vpvm krb5kdc[29134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261993, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:13.278861 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 29695
I20260809 07:53:13.278980 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:53:13.279264 29120 external_mini_cluster.cc:1428] Setting key be1d45133158ff1396e756dcbf9f25b8
I20260809 07:53:13.282506 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:13.271628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:36529 (local address 127.28.112.62:35757)
0809 07:53:13.271759 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:13.271763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:13.272524 (+ 761us) server_negotiation.cc:407] Connection header received
0809 07:53:13.273775 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:13.273779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:13.273838 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:13.273908 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:13.275050 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:13.275564 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:13.276665 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:13.276877 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:13.276965 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:13.279296 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:13.279299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:13.279301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:13.279329 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:13.280670 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:13.281121 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:13.281125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:13.281126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:13.281221 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:13.281563 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:13.281566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:13.281568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:13.281760 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:13.281837 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:13.282304 (+ 467us) server_negotiation.cc:299] Negotiation successful
0809 07:53:13.282395 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 07:53:13.283205 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:13.271970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:35757 (local address 127.28.112.3:36529)
0809 07:53:13.272381 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:13.272423 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:13.273572 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:13.274114 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:13.274122 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:13.274469 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:13.274906 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:13.274919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:13.275689 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:13.275692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:13.276537 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:13.276547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:13.277014 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:13.277019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:13.277181 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:13.277646 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:13.277662 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:13.279137 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:13.280771 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:13.280777 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:13.280791 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:13.281028 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:13.281311 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:13.281314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:13.281316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:13.281474 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:13.281850 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:13.281854 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:13.282109 (+ 255us) client_negotiation.cc:769] Sending connection context
0809 07:53:13.282254 (+ 145us) client_negotiation.cc:240] Negotiation successful
0809 07:53:13.282476 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":106,"threads_started":1}
I20260809 07:53:13.284162 29888 heartbeater.cc:347] Connected to a master server at 127.28.112.62:35757
I20260809 07:53:13.284366 29888 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:13.284797 29888 heartbeater.cc:511] Master 127.28.112.62:35757 requested a full tablet report, sending...
I20260809 07:53:13.285688 29209 ts_manager.cc:194] Registered new tserver with Master: 1185fd2a0fed46b4b9ff1be92271cc6b (127.28.112.3:42921)
I20260809 07:53:13.286242 29209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:36529
I20260809 07:53:13.293143 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 07:53:13 dist-test-slave-vpvm krb5kdc[29134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261993, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 07:53:13 dist-test-slave-vpvm krb5kdc[29134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261993, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.112.1@KRBTEST.COM
I20260809 07:53:13.317993 29897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:13.309650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35800 (local address 127.28.112.1:34141)
0809 07:53:13.310039 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:13.310049 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:13.310074 (+ 25us) server_negotiation.cc:407] Connection header received
0809 07:53:13.310124 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:13.310129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:13.310307 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:13.310424 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:13.310997 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:13.311764 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:13.312492 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:13.312847 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:13.312925 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:13.315099 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:13.315108 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:13.315123 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:13.315147 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:13.316470 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:13.316830 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:13.316835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:13.316836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:13.316931 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:13.317127 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:13.317129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:13.317130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:13.317349 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:13.317495 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:13.317631 (+ 136us) server_negotiation.cc:299] Negotiation successful
0809 07:53:13.317776 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":161,"threads_started":1}
W20260809 07:53:13.318925 29365 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35800
Aug 09 07:53:13 dist-test-slave-vpvm krb5kdc[29134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 07:53:13 dist-test-slave-vpvm krb5kdc[29134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.1@KRBTEST.COM
I20260809 07:53:13.343431 29897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:13.336009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35806 (local address 127.28.112.1:34141)
0809 07:53:13.336143 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:13.336147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:13.336266 (+ 119us) server_negotiation.cc:407] Connection header received
0809 07:53:13.336375 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:13.336377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:13.336419 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:13.336485 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:13.337074 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:13.337563 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:13.338411 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:13.338632 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:13.338717 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:13.340876 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:13.340879 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:13.340881 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:13.340904 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:13.342108 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:13.342447 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:13.342450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:13.342451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:13.342520 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:13.342707 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:13.342709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:13.342710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:13.342871 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:13.342976 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:13.343178 (+ 202us) server_negotiation.cc:299] Negotiation successful
0809 07:53:13.343275 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 07:53:13.344374 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 29300
I20260809 07:53:13.353986 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 29493
I20260809 07:53:13.361826 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 29695
I20260809 07:53:13.369815 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 29151
2026-08-09T07:53:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2246 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:53:13 dist-test-slave-vpvm krb5kdc[29909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:53:13 dist-test-slave-vpvm krb5kdc[29909](info): set up 2 sockets
Aug 09 07:53:13 dist-test-slave-vpvm krb5kdc[29909](info): commencing operation
krb5kdc: starting...
W20260809 07:53:14.506661 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:53:14 dist-test-slave-vpvm krb5kdc[29909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:53:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:53:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:53:14.618611 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:33897
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:40525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:33897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:14.735944 29931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:14.736243 29931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:14.736295 29931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:14.742085 29931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:14.742151 29931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:14.742177 29931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:14.742198 29931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:14.742224 29931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:53:14.747522 29931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:40525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:33897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:33897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:14.748485 29931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:14.749541 29931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:14.757508 29938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:14.757527 29940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:14.757619 29931 server_base.cc:1034] running on GCE node
W20260809 07:53:14.757527 29942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:14.758229 29931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:14.759193 29931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:14.760342 29931 hybrid_clock.cc:648] HybridClock initialized: now 1786261994760328 us; error 26 us; skew 500 ppm
Aug 09 07:53:14 dist-test-slave-vpvm krb5kdc[29909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261994, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:14.762850 29931 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:53:14.763731 29931 webserver.cc:460] Webserver started at http://127.28.112.62:42993/ using document root <none> and password file <none>
I20260809 07:53:14.764205 29931 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:14.764259 29931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:14.764415 29931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:14.766314 29931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "c300470c15214bce9e9cfbf1a41b8208"
format_stamp: "Formatted at 2026-08-09 07:53:14 on dist-test-slave-vpvm"
server_key: "b5ffe8f8cea4743fe5a4b76dcff4718f"
server_key_iv: "73a389f2753427415942781be7ec74c5"
server_key_version: "encryptionkey@0"
I20260809 07:53:14.766806 29931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "c300470c15214bce9e9cfbf1a41b8208"
format_stamp: "Formatted at 2026-08-09 07:53:14 on dist-test-slave-vpvm"
server_key: "b5ffe8f8cea4743fe5a4b76dcff4718f"
server_key_iv: "73a389f2753427415942781be7ec74c5"
server_key_version: "encryptionkey@0"
I20260809 07:53:14.770133 29931 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:53:14.772276 29951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:14.773375 29931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:14.773484 29931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "c300470c15214bce9e9cfbf1a41b8208"
format_stamp: "Formatted at 2026-08-09 07:53:14 on dist-test-slave-vpvm"
server_key: "b5ffe8f8cea4743fe5a4b76dcff4718f"
server_key_iv: "73a389f2753427415942781be7ec74c5"
server_key_version: "encryptionkey@0"
I20260809 07:53:14.773587 29931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:14.792469 29931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:14.794998 29931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:14.795156 29931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:14.801316 29931 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:33897
I20260809 07:53:14.801360 30027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:33897 every 8 connection(s)
I20260809 07:53:14.802191 29931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:53:14.803872 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 29931
I20260809 07:53:14.803984 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:53:14.804267 29120 external_mini_cluster.cc:1428] Setting key 9fd5c2d2e48e5e15cf8e9d47e5de5ba5
I20260809 07:53:14.805107 30028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:14.810530 30028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208: Bootstrap starting.
Aug 09 07:53:14 dist-test-slave-vpvm krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:14.812984 30028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:14.813669 30028 log.cc:824] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:14.815477 30028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208: No bootstrap required, opened a new log
I20260809 07:53:14.817123 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:14.805596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.28.112.62:33897)
0809 07:53:14.806022 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:14.806043 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:14.806078 (+ 35us) server_negotiation.cc:407] Connection header received
0809 07:53:14.806674 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:14.806688 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:14.806979 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:14.807277 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:14.808009 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:14.808951 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:14.809760 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:14.810251 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:14.810393 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:14.812899 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:14.812904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:14.812916 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:14.812943 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:14.814588 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:14.814980 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:14.814985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:14.814990 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:14.815102 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:14.815379 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:14.815383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:14.815384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:14.815777 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:14.815939 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:14.816225 (+ 286us) server_negotiation.cc:299] Negotiation successful
0809 07:53:14.816481 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":125,"threads_started":1}
I20260809 07:53:14.819150 30028 raft_consensus.cc:348] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c300470c15214bce9e9cfbf1a41b8208" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 33897 } }
I20260809 07:53:14.819408 30028 raft_consensus.cc:374] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:14.819468 30028 raft_consensus.cc:729] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c300470c15214bce9e9cfbf1a41b8208, State: Initialized, Role: FOLLOWER
I20260809 07:53:14.819872 30028 consensus_queue.cc:260] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c300470c15214bce9e9cfbf1a41b8208" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 33897 } }
I20260809 07:53:14.820003 30028 raft_consensus.cc:388] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:53:14.820070 30028 raft_consensus.cc:482] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:53:14.820166 30028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:14.820968 30028 raft_consensus.cc:504] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c300470c15214bce9e9cfbf1a41b8208" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 33897 } }
I20260809 07:53:14.821281 30028 leader_election.cc:302] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c300470c15214bce9e9cfbf1a41b8208; no voters:
I20260809 07:53:14.821560 30028 leader_election.cc:288] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:53:14.821712 30033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:14.821892 30033 raft_consensus.cc:686] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208 [term 1 LEADER]: Becoming Leader. State: Replica: c300470c15214bce9e9cfbf1a41b8208, State: Running, Role: LEADER
I20260809 07:53:14.822301 30033 consensus_queue.cc:237] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c300470c15214bce9e9cfbf1a41b8208" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 33897 } }
I20260809 07:53:14.822824 30028 sys_catalog.cc:564] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:14.824077 30036 sys_catalog.cc:455] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c300470c15214bce9e9cfbf1a41b8208" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c300470c15214bce9e9cfbf1a41b8208" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 33897 } } }
I20260809 07:53:14.824117 30037 sys_catalog.cc:455] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c300470c15214bce9e9cfbf1a41b8208. Latest consensus state: current_term: 1 leader_uuid: "c300470c15214bce9e9cfbf1a41b8208" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c300470c15214bce9e9cfbf1a41b8208" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 33897 } } }
I20260809 07:53:14.824213 30037 sys_catalog.cc:458] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:14.824211 30036 sys_catalog.cc:458] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:14.825167 30052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:53:14.827785 30052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:14.832765 30052 catalog_manager.cc:1347] Generated new cluster ID: eb2180bfda00499f9bc320a95d78a1c0
I20260809 07:53:14.832827 30052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:14.851738 30052 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:53:14.852558 30052 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:14.858981 30052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c300470c15214bce9e9cfbf1a41b8208: Generated new TSK 0
I20260809 07:53:14.859546 30052 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
I20260809 07:53:14.892680 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:33897
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:40525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:15.016956 30070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:15.017176 30070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:15.017216 30070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:15.021018 30070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:15.021061 30070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:15.021116 30070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:53:15.026021 30070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:40525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:33897
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:15.026875 30070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:15.027743 30070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:15.036535 30086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:15.036507 30081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:15.036567 30078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:15.036631 30070 server_base.cc:1034] running on GCE node
I20260809 07:53:15.037079 30070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:15.037693 30070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:15.038847 30070 hybrid_clock.cc:648] HybridClock initialized: now 1786261995038814 us; error 47 us; skew 500 ppm
Aug 09 07:53:15 dist-test-slave-vpvm krb5kdc[29909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261995, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:15.041368 30070 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:53:15.042212 30070 webserver.cc:460] Webserver started at http://127.28.112.1:46877/ using document root <none> and password file <none>
I20260809 07:53:15.042732 30070 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:15.042795 30070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:15.042990 30070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:15.044898 30070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "2d115119f3fa446db18f919deee65721"
format_stamp: "Formatted at 2026-08-09 07:53:15 on dist-test-slave-vpvm"
server_key: "53483f68b5f738771cd54f7f2b61118d"
server_key_iv: "a99f37b4d9e4f8dc58cff34999935cf7"
server_key_version: "encryptionkey@0"
I20260809 07:53:15.045404 30070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "2d115119f3fa446db18f919deee65721"
format_stamp: "Formatted at 2026-08-09 07:53:15 on dist-test-slave-vpvm"
server_key: "53483f68b5f738771cd54f7f2b61118d"
server_key_iv: "a99f37b4d9e4f8dc58cff34999935cf7"
server_key_version: "encryptionkey@0"
I20260809 07:53:15.049103 30070 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:53:15.051402 30094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:15.052474 30070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:15.052578 30070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "2d115119f3fa446db18f919deee65721"
format_stamp: "Formatted at 2026-08-09 07:53:15 on dist-test-slave-vpvm"
server_key: "53483f68b5f738771cd54f7f2b61118d"
server_key_iv: "a99f37b4d9e4f8dc58cff34999935cf7"
server_key_version: "encryptionkey@0"
I20260809 07:53:15.052672 30070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:15.090704 30070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:15.093331 30070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:15.093505 30070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:15.094126 30070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:15.094973 30070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:15.095022 30070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:15.095077 30070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:15.095106 30070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:15.103467 30070 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:36057
I20260809 07:53:15.103497 30268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:36057 every 8 connection(s)
I20260809 07:53:15.104305 30070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:53:15.108994 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 30070
I20260809 07:53:15.109086 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:53:15.109320 29120 external_mini_cluster.cc:1428] Setting key 796215429fdd125d36ff6555014b3ba7
Aug 09 07:53:15 dist-test-slave-vpvm krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261995, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:15.117372 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.106004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:39997 (local address 127.28.112.62:33897)
0809 07:53:15.106141 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:15.106145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:15.106939 (+ 794us) server_negotiation.cc:407] Connection header received
0809 07:53:15.107781 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:15.107785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:15.107841 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:15.107913 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:15.109028 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.109674 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.110989 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.111250 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.111334 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.114070 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:15.114073 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:15.114075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:15.114099 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:15.115361 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:15.115892 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:15.115895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:15.115896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:15.115974 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:15.116309 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:15.116312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:15.116313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:15.116491 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:15.116581 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:15.117070 (+ 489us) server_negotiation.cc:299] Negotiation successful
0809 07:53:15.117209 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 07:53:15.118203 30272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.106339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:33897 (local address 127.28.112.1:39997)
0809 07:53:15.106798 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:15.106837 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:15.107585 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:15.108089 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:15.108095 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:15.108417 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:15.108880 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:15.108893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.109833 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.109837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:15.110872 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:15.110885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.111409 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.111413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:15.111603 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.112175 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:15.112195 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:15.113895 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:15.115472 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:15.115478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:15.115493 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:15.115784 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:15.116056 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:15.116059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:15.116062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:15.116210 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:15.116593 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:15.116600 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:15.116830 (+ 230us) client_negotiation.cc:769] Sending connection context
0809 07:53:15.116990 (+ 160us) client_negotiation.cc:240] Negotiation successful
0809 07:53:15.117316 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.28.112.2@KRBTEST.COM; defaulting to no policy
I20260809 07:53:15.119369 30269 heartbeater.cc:347] Connected to a master server at 127.28.112.62:33897
I20260809 07:53:15.119596 30269 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:15.120107 30269 heartbeater.cc:511] Master 127.28.112.62:33897 requested a full tablet report, sending...
I20260809 07:53:15.121572 29979 ts_manager.cc:194] Registered new tserver with Master: 2d115119f3fa446db18f919deee65721 (127.28.112.1:36057)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.2@KRBTEST.COM" created.
I20260809 07:53:15.122751 29979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:39997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
I20260809 07:53:15.133818 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:33897
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:40525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:15.238773 30278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:15.238996 30278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:15.239043 30278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:15.242908 30278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:15.242960 30278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:15.243016 30278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:53:15.248049 30278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:40525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:33897
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:15.248987 30278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:15.249996 30278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:15.258940 30286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:15.258940 30283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:15.258944 30284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:15.259279 30278 server_base.cc:1034] running on GCE node
I20260809 07:53:15.259634 30278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:15.260175 30278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:15.261329 30278 hybrid_clock.cc:648] HybridClock initialized: now 1786261995261306 us; error 33 us; skew 500 ppm
Aug 09 07:53:15 dist-test-slave-vpvm krb5kdc[29909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261995, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:15.263837 30278 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:53:15.264619 30278 webserver.cc:460] Webserver started at http://127.28.112.2:39605/ using document root <none> and password file <none>
I20260809 07:53:15.265122 30278 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:15.265184 30278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:15.265377 30278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:15.267275 30278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "e0dca4fb2f274d45bfcee3b8dff5b21f"
format_stamp: "Formatted at 2026-08-09 07:53:15 on dist-test-slave-vpvm"
server_key: "d519eeb7c577db09b3fc00cad218a59c"
server_key_iv: "9870dcff261b3f725f3007fef300cf51"
server_key_version: "encryptionkey@0"
I20260809 07:53:15.267779 30278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "e0dca4fb2f274d45bfcee3b8dff5b21f"
format_stamp: "Formatted at 2026-08-09 07:53:15 on dist-test-slave-vpvm"
server_key: "d519eeb7c577db09b3fc00cad218a59c"
server_key_iv: "9870dcff261b3f725f3007fef300cf51"
server_key_version: "encryptionkey@0"
I20260809 07:53:15.271320 30278 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:53:15.273438 30300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:15.274536 30278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:15.274641 30278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "e0dca4fb2f274d45bfcee3b8dff5b21f"
format_stamp: "Formatted at 2026-08-09 07:53:15 on dist-test-slave-vpvm"
server_key: "d519eeb7c577db09b3fc00cad218a59c"
server_key_iv: "9870dcff261b3f725f3007fef300cf51"
server_key_version: "encryptionkey@0"
I20260809 07:53:15.274724 30278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:15.283303 30278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:15.285640 30278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:15.285789 30278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:15.286335 30278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:15.287068 30278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:15.287113 30278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:15.287160 30278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:15.287187 30278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:15.295464 30278 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:43505
I20260809 07:53:15.295492 30466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:43505 every 8 connection(s)
I20260809 07:53:15.296242 30278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:53:15.298818 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 30278
I20260809 07:53:15.298945 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:53:15.299242 29120 external_mini_cluster.cc:1428] Setting key ff33c49def5df12399d62ae0f8328fb6
Aug 09 07:53:15 dist-test-slave-vpvm krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261995, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:15.309441 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.297893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:38957 (local address 127.28.112.62:33897)
0809 07:53:15.298071 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:15.298075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:15.298951 (+ 876us) server_negotiation.cc:407] Connection header received
0809 07:53:15.300042 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:15.300045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:15.300099 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:15.300174 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:15.301598 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.302146 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.303358 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.303576 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.303672 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.306239 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:15.306242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:15.306244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:15.306267 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:15.307492 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:15.308058 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:15.308060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:15.308061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:15.308126 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:15.308461 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:15.308463 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:15.308464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:15.308616 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:15.308685 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:15.309234 (+ 549us) server_negotiation.cc:299] Negotiation successful
0809 07:53:15.309325 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.28.112.3@KRBTEST.COM; defaulting to no policy
I20260809 07:53:15.310499 30471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.298293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:33897 (local address 127.28.112.2:38957)
0809 07:53:15.298791 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:15.298849 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:15.299834 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:15.300356 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:15.300365 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:15.300820 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:15.301443 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:15.301463 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.302275 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.302280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:15.303236 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:15.303246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.303718 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.303722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:15.303890 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.304394 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:15.304410 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:15.306080 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:15.307604 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:15.307610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:15.307625 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:15.307971 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:15.308202 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:15.308205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:15.308207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:15.308380 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:15.308710 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:15.308717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:15.309005 (+ 288us) client_negotiation.cc:769] Sending connection context
0809 07:53:15.309222 (+ 217us) client_negotiation.cc:240] Negotiation successful
0809 07:53:15.309543 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":106,"threads_started":1}
I20260809 07:53:15.311547 30469 heartbeater.cc:347] Connected to a master server at 127.28.112.62:33897
I20260809 07:53:15.311756 30469 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:15.312256 30469 heartbeater.cc:511] Master 127.28.112.62:33897 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.3@KRBTEST.COM" created.
I20260809 07:53:15.313342 29979 ts_manager.cc:194] Registered new tserver with Master: e0dca4fb2f274d45bfcee3b8dff5b21f (127.28.112.2:43505)
I20260809 07:53:15.313892 29979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:38957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
I20260809 07:53:15.328284 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:33897
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:40525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:15.435627 30475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:15.435849 30475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:15.435907 30475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:15.439765 30475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:15.439814 30475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:15.439890 30475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:53:15.444842 30475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:40525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:33897
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:15.445755 30475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:15.446698 30475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:15.455705 30485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:15.455730 30483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:15.455775 30475 server_base.cc:1034] running on GCE node
W20260809 07:53:15.455981 30487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:15.456329 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:15.456944 30475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:15.458097 30475 hybrid_clock.cc:648] HybridClock initialized: now 1786261995458082 us; error 25 us; skew 500 ppm
Aug 09 07:53:15 dist-test-slave-vpvm krb5kdc[29909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261995, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:15.460554 30475 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:53:15.461380 30475 webserver.cc:460] Webserver started at http://127.28.112.3:43947/ using document root <none> and password file <none>
I20260809 07:53:15.461927 30475 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:15.462102 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:15.462335 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:15.464311 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "14f90110c88c4f438e9c05eadf709938"
format_stamp: "Formatted at 2026-08-09 07:53:15 on dist-test-slave-vpvm"
server_key: "04aa6e06ec15f5465fc1b8307f30190f"
server_key_iv: "218dfc5826d17ef425f8b9c8c169fc4c"
server_key_version: "encryptionkey@0"
I20260809 07:53:15.464851 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "14f90110c88c4f438e9c05eadf709938"
format_stamp: "Formatted at 2026-08-09 07:53:15 on dist-test-slave-vpvm"
server_key: "04aa6e06ec15f5465fc1b8307f30190f"
server_key_iv: "218dfc5826d17ef425f8b9c8c169fc4c"
server_key_version: "encryptionkey@0"
I20260809 07:53:15.468271 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 07:53:15.470460 30496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:15.471550 30475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:15.471657 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "14f90110c88c4f438e9c05eadf709938"
format_stamp: "Formatted at 2026-08-09 07:53:15 on dist-test-slave-vpvm"
server_key: "04aa6e06ec15f5465fc1b8307f30190f"
server_key_iv: "218dfc5826d17ef425f8b9c8c169fc4c"
server_key_version: "encryptionkey@0"
I20260809 07:53:15.471740 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:15.490065 30475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:15.492451 30475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:15.492607 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:15.493125 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:15.493860 30475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:15.493904 30475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:15.493984 30475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:15.494014 30475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:15.502316 30475 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:41269
I20260809 07:53:15.502362 30665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:41269 every 8 connection(s)
I20260809 07:53:15.503131 30475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:53:15.503640 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 30475
I20260809 07:53:15.503751 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:53:15.504024 29120 external_mini_cluster.cc:1428] Setting key 2e80442cc63fdf6c75eb921a551a3325
Aug 09 07:53:15 dist-test-slave-vpvm krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261995, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:15.515113 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.504838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:47191 (local address 127.28.112.62:33897)
0809 07:53:15.504954 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:15.504959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:15.505914 (+ 955us) server_negotiation.cc:407] Connection header received
0809 07:53:15.506830 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:15.506833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:15.506886 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:15.506956 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:15.508002 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.508520 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.509566 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.509785 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.509871 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.512060 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:15.512064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:15.512066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:15.512088 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:15.513200 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:15.513652 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:15.513655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:15.513657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:15.513733 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:15.514055 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:15.514058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:15.514060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:15.514231 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:15.514318 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:15.514829 (+ 511us) server_negotiation.cc:299] Negotiation successful
0809 07:53:15.514944 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 07:53:15.515798 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.505324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:33897 (local address 127.28.112.3:47191)
0809 07:53:15.505790 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:15.505828 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:15.506649 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:15.507113 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:15.507119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:15.507431 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:15.507875 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:15.507887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.508634 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.508637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:15.509453 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:15.509462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.509914 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.509918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:15.510107 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.510598 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:15.510613 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:15.511905 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:15.513303 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:15.513309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:15.513323 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:15.513561 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:15.513814 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:15.513817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:15.513819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:15.513942 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:15.514329 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:15.514335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:15.514560 (+ 225us) client_negotiation.cc:769] Sending connection context
0809 07:53:15.514749 (+ 189us) client_negotiation.cc:240] Negotiation successful
0809 07:53:15.515047 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":130,"threads_started":1}
I20260809 07:53:15.516805 30666 heartbeater.cc:347] Connected to a master server at 127.28.112.62:33897
I20260809 07:53:15.517028 30666 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:15.517503 30666 heartbeater.cc:511] Master 127.28.112.62:33897 requested a full tablet report, sending...
I20260809 07:53:15.518465 29979 ts_manager.cc:194] Registered new tserver with Master: 14f90110c88c4f438e9c05eadf709938 (127.28.112.3:41269)
I20260809 07:53:15.518975 29979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:47191
I20260809 07:53:15.529282 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 07:53:15.538940 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.532454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.28.112.62:33897)
0809 07:53:15.532557 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:15.532561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:15.532748 (+ 187us) server_negotiation.cc:407] Connection header received
0809 07:53:15.532853 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:15.532856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:15.532896 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:15.532957 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:15.533667 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.534226 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.535011 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.535218 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.535302 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.536468 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:15.536471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:15.536473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:15.536495 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:15.537450 (+ 955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:15.537820 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:15.537824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:15.537825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:15.537897 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:15.538157 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:15.538160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:15.538162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:15.538356 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:15.538472 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:15.538703 (+ 231us) server_negotiation.cc:299] Negotiation successful
0809 07:53:15.538805 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 07:53:15.543004 29979 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:53:15.545799 29979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:53:15.559307 30687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.554234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:41269 (local address 127.0.0.1:53408)
0809 07:53:15.554627 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:15.554639 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:15.554726 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:15.555427 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:15.555430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:15.555447 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:15.555542 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:15.555548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.556859 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.556862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:15.558007 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:15.558015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.558791 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.558795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:15.558904 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.558941 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 07:53:15.559055 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 07:53:15.559167 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":1,"thread_start_us":121,"threads_started":1}
I20260809 07:53:15.559361 30690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.554584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53408 (local address 127.28.112.3:41269)
0809 07:53:15.555040 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:15.555052 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:15.555074 (+ 22us) server_negotiation.cc:407] Connection header received
0809 07:53:15.555140 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:15.555145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:15.555321 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:15.555458 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:15.555628 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.556707 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.558112 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.558679 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.558785 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.558925 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:15.559102 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 07:53:15.559209 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":156,"threads_started":1}
I20260809 07:53:15.559625 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.553705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55568 (local address 127.28.112.1:36057)
0809 07:53:15.554601 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:15.554615 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:15.554652 (+ 37us) server_negotiation.cc:407] Connection header received
0809 07:53:15.554732 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:15.554737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:15.554943 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:15.555102 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:15.555391 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.556380 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.557988 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.558780 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.558879 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.559016 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:15.559378 (+ 362us) server_negotiation.cc:299] Negotiation successful
0809 07:53:15.559470 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":83,"threads_started":1}
I20260809 07:53:15.559703 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.553570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:36057 (local address 127.0.0.1:55568)
0809 07:53:15.553838 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:15.553879 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:15.554022 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:15.555117 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:15.555123 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:15.555148 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:15.555278 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:15.555284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.556497 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.556500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:15.557842 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:15.557853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.559077 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.559081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:15.559168 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.559188 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 07:53:15.559252 (+ 64us) client_negotiation.cc:240] Negotiation successful
0809 07:53:15.559305 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":55,"threads_started":1}
I20260809 07:53:15.560189 30686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.553782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:43505 (local address 127.0.0.1:33594)
0809 07:53:15.554266 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:15.554279 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:15.554375 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:15.555751 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:15.555754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:15.555774 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:15.555870 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:15.555876 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.557240 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.557243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:15.558484 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:15.558495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.559818 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.559822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:15.559907 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.559929 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 07:53:15.560005 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 07:53:15.560068 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":24,"thread_start_us":81,"threads_started":1}
I20260809 07:53:15.560285 30688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.554202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33594 (local address 127.28.112.2:43505)
0809 07:53:15.555170 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:15.555182 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:15.555204 (+ 22us) server_negotiation.cc:407] Connection header received
0809 07:53:15.555282 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:15.555287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:15.555452 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:15.555592 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:15.555998 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.557103 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.558618 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.559361 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.559604 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.559753 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:15.560051 (+ 298us) server_negotiation.cc:299] Negotiation successful
0809 07:53:15.560127 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":153,"threads_started":1}
I20260809 07:53:15.562732 30169 tablet_service.cc:1467] Processing CreateTablet for tablet 44edd26fc37a489499deb24a5f642ad5 (DEFAULT_TABLE table=test-table [id=221cca8763ab462189daf41035502b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:15.562767 30373 tablet_service.cc:1467] Processing CreateTablet for tablet 44edd26fc37a489499deb24a5f642ad5 (DEFAULT_TABLE table=test-table [id=221cca8763ab462189daf41035502b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:15.562767 30568 tablet_service.cc:1467] Processing CreateTablet for tablet 44edd26fc37a489499deb24a5f642ad5 (DEFAULT_TABLE table=test-table [id=221cca8763ab462189daf41035502b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:15.563705 30169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44edd26fc37a489499deb24a5f642ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:15.563803 30373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44edd26fc37a489499deb24a5f642ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:15.563803 30568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44edd26fc37a489499deb24a5f642ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:15.569805 30692 tablet_bootstrap.cc:492] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721: Bootstrap starting.
I20260809 07:53:15.570901 30696 tablet_bootstrap.cc:492] T 44edd26fc37a489499deb24a5f642ad5 P e0dca4fb2f274d45bfcee3b8dff5b21f: Bootstrap starting.
I20260809 07:53:15.571151 30695 tablet_bootstrap.cc:492] T 44edd26fc37a489499deb24a5f642ad5 P 14f90110c88c4f438e9c05eadf709938: Bootstrap starting.
I20260809 07:53:15.572139 30692 tablet_bootstrap.cc:654] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:15.572975 30695 tablet_bootstrap.cc:654] T 44edd26fc37a489499deb24a5f642ad5 P 14f90110c88c4f438e9c05eadf709938: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:15.573387 30696 tablet_bootstrap.cc:654] T 44edd26fc37a489499deb24a5f642ad5 P e0dca4fb2f274d45bfcee3b8dff5b21f: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:15.573505 30692 log.cc:824] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:15.573995 30695 log.cc:824] T 44edd26fc37a489499deb24a5f642ad5 P 14f90110c88c4f438e9c05eadf709938: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:15.574185 30696 log.cc:824] T 44edd26fc37a489499deb24a5f642ad5 P e0dca4fb2f274d45bfcee3b8dff5b21f: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:15.575623 30692 tablet_bootstrap.cc:492] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721: No bootstrap required, opened a new log
I20260809 07:53:15.575850 30692 ts_tablet_manager.cc:1397] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260809 07:53:15.575910 30696 tablet_bootstrap.cc:492] T 44edd26fc37a489499deb24a5f642ad5 P e0dca4fb2f274d45bfcee3b8dff5b21f: No bootstrap required, opened a new log
I20260809 07:53:15.575958 30695 tablet_bootstrap.cc:492] T 44edd26fc37a489499deb24a5f642ad5 P 14f90110c88c4f438e9c05eadf709938: No bootstrap required, opened a new log
I20260809 07:53:15.576099 30696 ts_tablet_manager.cc:1397] T 44edd26fc37a489499deb24a5f642ad5 P e0dca4fb2f274d45bfcee3b8dff5b21f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 07:53:15.576128 30695 ts_tablet_manager.cc:1397] T 44edd26fc37a489499deb24a5f642ad5 P 14f90110c88c4f438e9c05eadf709938: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 07:53:15.579074 30696 raft_consensus.cc:348] T 44edd26fc37a489499deb24a5f642ad5 P e0dca4fb2f274d45bfcee3b8dff5b21f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d115119f3fa446db18f919deee65721" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36057 } } peers { permanent_uuid: "14f90110c88c4f438e9c05eadf709938" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41269 } } peers { permanent_uuid: "e0dca4fb2f274d45bfcee3b8dff5b21f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43505 } }
I20260809 07:53:15.579293 30696 raft_consensus.cc:374] T 44edd26fc37a489499deb24a5f642ad5 P e0dca4fb2f274d45bfcee3b8dff5b21f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:15.579326 30696 raft_consensus.cc:729] T 44edd26fc37a489499deb24a5f642ad5 P e0dca4fb2f274d45bfcee3b8dff5b21f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0dca4fb2f274d45bfcee3b8dff5b21f, State: Initialized, Role: FOLLOWER
I20260809 07:53:15.579685 30696 consensus_queue.cc:260] T 44edd26fc37a489499deb24a5f642ad5 P e0dca4fb2f274d45bfcee3b8dff5b21f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d115119f3fa446db18f919deee65721" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36057 } } peers { permanent_uuid: "14f90110c88c4f438e9c05eadf709938" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41269 } } peers { permanent_uuid: "e0dca4fb2f274d45bfcee3b8dff5b21f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43505 } }
I20260809 07:53:15.579716 30692 raft_consensus.cc:348] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d115119f3fa446db18f919deee65721" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36057 } } peers { permanent_uuid: "14f90110c88c4f438e9c05eadf709938" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41269 } } peers { permanent_uuid: "e0dca4fb2f274d45bfcee3b8dff5b21f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43505 } }
I20260809 07:53:15.579716 30695 raft_consensus.cc:348] T 44edd26fc37a489499deb24a5f642ad5 P 14f90110c88c4f438e9c05eadf709938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d115119f3fa446db18f919deee65721" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36057 } } peers { permanent_uuid: "14f90110c88c4f438e9c05eadf709938" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41269 } } peers { permanent_uuid: "e0dca4fb2f274d45bfcee3b8dff5b21f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43505 } }
I20260809 07:53:15.579953 30692 raft_consensus.cc:374] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:15.579953 30695 raft_consensus.cc:374] T 44edd26fc37a489499deb24a5f642ad5 P 14f90110c88c4f438e9c05eadf709938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:15.580003 30692 raft_consensus.cc:729] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d115119f3fa446db18f919deee65721, State: Initialized, Role: FOLLOWER
I20260809 07:53:15.580003 30695 raft_consensus.cc:729] T 44edd26fc37a489499deb24a5f642ad5 P 14f90110c88c4f438e9c05eadf709938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f90110c88c4f438e9c05eadf709938, State: Initialized, Role: FOLLOWER
I20260809 07:53:15.580256 30469 heartbeater.cc:503] Master 127.28.112.62:33897 was elected leader, sending a full tablet report...
I20260809 07:53:15.580425 30695 consensus_queue.cc:260] T 44edd26fc37a489499deb24a5f642ad5 P 14f90110c88c4f438e9c05eadf709938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d115119f3fa446db18f919deee65721" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36057 } } peers { permanent_uuid: "14f90110c88c4f438e9c05eadf709938" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41269 } } peers { permanent_uuid: "e0dca4fb2f274d45bfcee3b8dff5b21f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43505 } }
I20260809 07:53:15.580425 30692 consensus_queue.cc:260] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d115119f3fa446db18f919deee65721" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36057 } } peers { permanent_uuid: "14f90110c88c4f438e9c05eadf709938" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41269 } } peers { permanent_uuid: "e0dca4fb2f274d45bfcee3b8dff5b21f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43505 } }
I20260809 07:53:15.580590 30696 ts_tablet_manager.cc:1428] T 44edd26fc37a489499deb24a5f642ad5 P e0dca4fb2f274d45bfcee3b8dff5b21f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260809 07:53:15.581010 30666 heartbeater.cc:503] Master 127.28.112.62:33897 was elected leader, sending a full tablet report...
I20260809 07:53:15.581360 30692 ts_tablet_manager.cc:1428] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 07:53:15.581360 30695 ts_tablet_manager.cc:1428] T 44edd26fc37a489499deb24a5f642ad5 P 14f90110c88c4f438e9c05eadf709938: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 07:53:15.581668 30269 heartbeater.cc:503] Master 127.28.112.62:33897 was elected leader, sending a full tablet report...
W20260809 07:53:15.605507 30271 tablet.cc:2369] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:53:15.733036 30707 raft_consensus.cc:482] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:15.733250 30707 raft_consensus.cc:504] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d115119f3fa446db18f919deee65721" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36057 } } peers { permanent_uuid: "14f90110c88c4f438e9c05eadf709938" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41269 } } peers { permanent_uuid: "e0dca4fb2f274d45bfcee3b8dff5b21f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43505 } }
I20260809 07:53:15.734426 30707 leader_election.cc:288] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14f90110c88c4f438e9c05eadf709938 (127.28.112.3:41269), e0dca4fb2f274d45bfcee3b8dff5b21f (127.28.112.2:43505)
I20260809 07:53:15.739538 30688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.734829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:41033 (local address 127.28.112.2:43505)
0809 07:53:15.735009 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:15.735012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:15.735101 (+ 89us) server_negotiation.cc:407] Connection header received
0809 07:53:15.735248 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:15.735251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:15.735304 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:15.735375 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:15.735764 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.736534 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.738297 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.738911 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.739031 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.739085 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:15.739340 (+ 255us) server_negotiation.cc:299] Negotiation successful
0809 07:53:15.739405 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260809 07:53:15.739552 30690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.734764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:41793 (local address 127.28.112.3:41269)
0809 07:53:15.734902 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:15.734907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:15.735095 (+ 188us) server_negotiation.cc:407] Connection header received
0809 07:53:15.735248 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:15.735251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:15.735300 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:15.735379 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:15.735763 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.736534 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.738297 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.738912 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.739030 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.739085 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:15.739340 (+ 255us) server_negotiation.cc:299] Negotiation successful
0809 07:53:15.739405 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 07:53:15.739601 30709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.734697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:43505 (local address 127.28.112.1:41033)
0809 07:53:15.735013 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:15.735025 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:15.735144 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:15.735398 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:15.735400 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:15.735426 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:15.735658 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:15.735666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.736658 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.736660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:15.738169 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:15.738181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.739027 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.739031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:15.739192 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.739244 (+ 52us) client_negotiation.cc:769] Sending connection context
0809 07:53:15.739320 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 07:53:15.739424 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":187,"spinlock_wait_cycles":32384,"thread_start_us":124,"threads_started":1}
I20260809 07:53:15.739612 30708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.734645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:41269 (local address 127.28.112.1:41793)
0809 07:53:15.734991 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:15.735006 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:15.735152 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:15.735398 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:15.735401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:15.735424 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:15.735664 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:15.735670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.736650 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.736654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:15.738169 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:15.738181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.739027 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.739031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:15.739203 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.739244 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 07:53:15.739321 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 07:53:15.739424 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":275,"spinlock_wait_cycles":32896,"thread_start_us":98,"threads_started":1}
I20260809 07:53:15.740501 30400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44edd26fc37a489499deb24a5f642ad5" candidate_uuid: "2d115119f3fa446db18f919deee65721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0dca4fb2f274d45bfcee3b8dff5b21f" is_pre_election: true
I20260809 07:53:15.740501 30600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44edd26fc37a489499deb24a5f642ad5" candidate_uuid: "2d115119f3fa446db18f919deee65721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f90110c88c4f438e9c05eadf709938" is_pre_election: true
I20260809 07:53:15.740798 30400 raft_consensus.cc:2454] T 44edd26fc37a489499deb24a5f642ad5 P e0dca4fb2f274d45bfcee3b8dff5b21f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d115119f3fa446db18f919deee65721 in term 0.
I20260809 07:53:15.740803 30600 raft_consensus.cc:2454] T 44edd26fc37a489499deb24a5f642ad5 P 14f90110c88c4f438e9c05eadf709938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d115119f3fa446db18f919deee65721 in term 0.
I20260809 07:53:15.741290 30098 leader_election.cc:302] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d115119f3fa446db18f919deee65721, e0dca4fb2f274d45bfcee3b8dff5b21f; no voters:
I20260809 07:53:15.741565 30707 raft_consensus.cc:2781] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:53:15.741657 30707 raft_consensus.cc:482] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:53:15.741707 30707 raft_consensus.cc:3037] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:15.742733 30707 raft_consensus.cc:504] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d115119f3fa446db18f919deee65721" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36057 } } peers { permanent_uuid: "14f90110c88c4f438e9c05eadf709938" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41269 } } peers { permanent_uuid: "e0dca4fb2f274d45bfcee3b8dff5b21f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43505 } }
I20260809 07:53:15.743032 30707 leader_election.cc:288] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721 [CANDIDATE]: Term 1 election: Requested vote from peers 14f90110c88c4f438e9c05eadf709938 (127.28.112.3:41269), e0dca4fb2f274d45bfcee3b8dff5b21f (127.28.112.2:43505)
I20260809 07:53:15.743346 30600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44edd26fc37a489499deb24a5f642ad5" candidate_uuid: "2d115119f3fa446db18f919deee65721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f90110c88c4f438e9c05eadf709938"
I20260809 07:53:15.743407 30400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44edd26fc37a489499deb24a5f642ad5" candidate_uuid: "2d115119f3fa446db18f919deee65721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0dca4fb2f274d45bfcee3b8dff5b21f"
I20260809 07:53:15.743458 30600 raft_consensus.cc:3037] T 44edd26fc37a489499deb24a5f642ad5 P 14f90110c88c4f438e9c05eadf709938 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:15.743521 30400 raft_consensus.cc:3037] T 44edd26fc37a489499deb24a5f642ad5 P e0dca4fb2f274d45bfcee3b8dff5b21f [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:15.744467 30600 raft_consensus.cc:2454] T 44edd26fc37a489499deb24a5f642ad5 P 14f90110c88c4f438e9c05eadf709938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d115119f3fa446db18f919deee65721 in term 1.
I20260809 07:53:15.744486 30400 raft_consensus.cc:2454] T 44edd26fc37a489499deb24a5f642ad5 P e0dca4fb2f274d45bfcee3b8dff5b21f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d115119f3fa446db18f919deee65721 in term 1.
I20260809 07:53:15.744764 30099 leader_election.cc:302] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14f90110c88c4f438e9c05eadf709938, 2d115119f3fa446db18f919deee65721; no voters:
I20260809 07:53:15.744979 30707 raft_consensus.cc:2781] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:15.745188 30707 raft_consensus.cc:686] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721 [term 1 LEADER]: Becoming Leader. State: Replica: 2d115119f3fa446db18f919deee65721, State: Running, Role: LEADER
I20260809 07:53:15.745517 30707 consensus_queue.cc:237] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d115119f3fa446db18f919deee65721" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36057 } } peers { permanent_uuid: "14f90110c88c4f438e9c05eadf709938" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41269 } } peers { permanent_uuid: "e0dca4fb2f274d45bfcee3b8dff5b21f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43505 } }
I20260809 07:53:15.748494 29979 catalog_manager.cc:5495] T 44edd26fc37a489499deb24a5f642ad5 P 2d115119f3fa446db18f919deee65721 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d115119f3fa446db18f919deee65721 (127.28.112.1). New cstate: current_term: 1 leader_uuid: "2d115119f3fa446db18f919deee65721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d115119f3fa446db18f919deee65721" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14f90110c88c4f438e9c05eadf709938" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0dca4fb2f274d45bfcee3b8dff5b21f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43505 } health_report { overall_health: UNKNOWN } } }
W20260809 07:53:15.753981 30667 tablet.cc:2369] T 44edd26fc37a489499deb24a5f642ad5 P 14f90110c88c4f438e9c05eadf709938: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 07:53:15 dist-test-slave-vpvm krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.1@KRBTEST.COM
I20260809 07:53:15.792263 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.784127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55576 (local address 127.28.112.1:36057)
0809 07:53:15.784287 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:15.784291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:15.784419 (+ 128us) server_negotiation.cc:407] Connection header received
0809 07:53:15.784555 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:15.784558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:15.784606 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:15.784674 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:15.785386 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.785897 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.786740 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.786963 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.787050 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.789366 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:15.789377 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:15.789394 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:15.789419 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:15.790840 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:15.791210 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:15.791215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:15.791216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:15.791289 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:15.791469 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:15.791471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:15.791473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:15.791727 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:15.791877 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:15.792005 (+ 128us) server_negotiation.cc:299] Negotiation successful
0809 07:53:15.792102 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260809 07:53:15.797308 30470 tablet.cc:2369] T 44edd26fc37a489499deb24a5f642ad5 P e0dca4fb2f274d45bfcee3b8dff5b21f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 07:53:15 dist-test-slave-vpvm krb5kdc[29909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261995, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 07:53:15 dist-test-slave-vpvm krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261995, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.112.1@KRBTEST.COM
I20260809 07:53:15.820278 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.812113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55580 (local address 127.28.112.1:36057)
0809 07:53:15.812257 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:15.812262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:15.812404 (+ 142us) server_negotiation.cc:407] Connection header received
0809 07:53:15.812628 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:15.812633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:15.812693 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:15.812776 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:15.813562 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.814114 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.814968 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.815178 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.815264 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.817533 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:15.817542 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:15.817545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:15.817569 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:15.818751 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:15.819134 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:15.819137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:15.819138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:15.819210 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:15.819448 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:15.819451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:15.819453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:15.819624 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:15.819761 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:15.820016 (+ 255us) server_negotiation.cc:299] Negotiation successful
0809 07:53:15.820131 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Aug 09 07:53:15 dist-test-slave-vpvm krb5kdc[29909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 07:53:15 dist-test-slave-vpvm krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.1@KRBTEST.COM
I20260809 07:53:15.845188 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:15.837660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55582 (local address 127.28.112.1:36057)
0809 07:53:15.837806 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:15.837812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:15.837943 (+ 131us) server_negotiation.cc:407] Connection header received
0809 07:53:15.838074 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:15.838077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:15.838121 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:15.838196 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:15.838810 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.839324 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.840164 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:15.840381 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:15.840475 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:15.842756 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:15.842758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:15.842761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:15.842783 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:15.843936 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:15.844285 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:15.844287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:15.844288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:15.844351 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:15.844558 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:15.844559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:15.844560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:15.844705 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:15.844808 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:15.844946 (+ 138us) server_negotiation.cc:299] Negotiation successful
0809 07:53:15.845034 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260809 07:53:15.846552 30132 tablet_service.cc:2908] Rejecting scan request for tablet 44edd26fc37a489499deb24a5f642ad5: Uninitialized: clean time has not yet been initialized
I20260809 07:53:15.848493 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 30070
I20260809 07:53:15.863663 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 30278
I20260809 07:53:15.872256 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 30475
I20260809 07:53:15.882251 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 29931
2026-08-09T07:53:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2514 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:53:15 dist-test-slave-vpvm krb5kdc[30730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:53:15 dist-test-slave-vpvm krb5kdc[30730](info): set up 2 sockets
Aug 09 07:53:15 dist-test-slave-vpvm krb5kdc[30730](info): commencing operation
krb5kdc: starting...
W20260809 07:53:17.016882 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:53:17 dist-test-slave-vpvm krb5kdc[30730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:53:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:53:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:53:17.128489 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:43097
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:43097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:17.236372 30748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:17.236603 30748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:17.236666 30748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:17.240641 30748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:17.240694 30748 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 07:53:17.240718 30748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:17.240742 30748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:17.240774 30748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:17.240803 30748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:53:17.245848 30748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:43097
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:43097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:17.246804 30748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:17.247747 30748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:17.255796 30756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:17.255829 30755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:17.255940 30748 server_base.cc:1034] running on GCE node
W20260809 07:53:17.255800 30759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:17.256613 30748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:17.257615 30748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:17.258765 30748 hybrid_clock.cc:648] HybridClock initialized: now 1786261997258750 us; error 25 us; skew 500 ppm
Aug 09 07:53:17 dist-test-slave-vpvm krb5kdc[30730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261997, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:17.264504 30748 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:53:17.265336 30748 webserver.cc:460] Webserver started at http://127.28.112.62:34845/ using document root <none> and password file <none>
I20260809 07:53:17.265838 30748 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:17.265903 30748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:17.266124 30748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:17.268018 30748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "34b47348a96a4951b8128f7e8e7d6264"
format_stamp: "Formatted at 2026-08-09 07:53:17 on dist-test-slave-vpvm"
server_key: "44caa3e64fe63c714e35e480f6064c90"
server_key_iv: "177b2cb57da0112d0930c1e249f563d1"
server_key_version: "encryptionkey@0"
I20260809 07:53:17.268532 30748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "34b47348a96a4951b8128f7e8e7d6264"
format_stamp: "Formatted at 2026-08-09 07:53:17 on dist-test-slave-vpvm"
server_key: "44caa3e64fe63c714e35e480f6064c90"
server_key_iv: "177b2cb57da0112d0930c1e249f563d1"
server_key_version: "encryptionkey@0"
I20260809 07:53:17.272063 30748 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 07:53:17.274358 30776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:17.275481 30748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:17.275591 30748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "34b47348a96a4951b8128f7e8e7d6264"
format_stamp: "Formatted at 2026-08-09 07:53:17 on dist-test-slave-vpvm"
server_key: "44caa3e64fe63c714e35e480f6064c90"
server_key_iv: "177b2cb57da0112d0930c1e249f563d1"
server_key_version: "encryptionkey@0"
I20260809 07:53:17.275678 30748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:17.291500 30748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:17.293990 30748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:17.294163 30748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:17.301299 30748 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:43097
I20260809 07:53:17.301312 30857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:43097 every 8 connection(s)
I20260809 07:53:17.302198 30748 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:53:17.303645 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 30748
I20260809 07:53:17.303750 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:53:17.303982 29120 external_mini_cluster.cc:1428] Setting key 6ee089cc65cc165b641fceaadc2c66ba
I20260809 07:53:17.305080 30859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:17.311349 30859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264: Bootstrap starting.
Aug 09 07:53:17 dist-test-slave-vpvm krb5kdc[30730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:17.313771 30859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:17.315076 30859 log.cc:824] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:17.316949 30859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264: No bootstrap required, opened a new log
I20260809 07:53:17.317283 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:17.305193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58362 (local address 127.28.112.62:43097)
0809 07:53:17.305561 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:17.305576 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:17.305611 (+ 35us) server_negotiation.cc:407] Connection header received
0809 07:53:17.306270 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:17.306288 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:17.306592 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:17.306849 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:17.307700 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.308672 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.309784 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.310262 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.310372 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:17.312985 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:17.312990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:17.313003 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:17.313029 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:17.314544 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:17.315012 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:17.315016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:17.315020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:17.315131 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:17.315397 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:17.315400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:17.315402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:17.315876 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:17.316061 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:17.316334 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 07:53:17.316593 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260809 07:53:17.320243 30859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b47348a96a4951b8128f7e8e7d6264" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43097 } }
I20260809 07:53:17.320449 30859 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:17.320520 30859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b47348a96a4951b8128f7e8e7d6264, State: Initialized, Role: FOLLOWER
I20260809 07:53:17.320904 30859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b47348a96a4951b8128f7e8e7d6264" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43097 } }
I20260809 07:53:17.321030 30859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:53:17.321094 30859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:53:17.321193 30859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:17.322017 30859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b47348a96a4951b8128f7e8e7d6264" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43097 } }
I20260809 07:53:17.322333 30859 leader_election.cc:302] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34b47348a96a4951b8128f7e8e7d6264; no voters:
I20260809 07:53:17.322602 30859 leader_election.cc:288] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:53:17.322702 30869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:17.322891 30869 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264 [term 1 LEADER]: Becoming Leader. State: Replica: 34b47348a96a4951b8128f7e8e7d6264, State: Running, Role: LEADER
I20260809 07:53:17.323220 30869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b47348a96a4951b8128f7e8e7d6264" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43097 } }
I20260809 07:53:17.323758 30859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:17.324687 30873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34b47348a96a4951b8128f7e8e7d6264. Latest consensus state: current_term: 1 leader_uuid: "34b47348a96a4951b8128f7e8e7d6264" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b47348a96a4951b8128f7e8e7d6264" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43097 } } }
I20260809 07:53:17.324841 30873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:17.325204 30884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:53:17.325479 30872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34b47348a96a4951b8128f7e8e7d6264" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b47348a96a4951b8128f7e8e7d6264" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43097 } } }
I20260809 07:53:17.325626 30872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:17.328172 30884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:17.333516 30884 catalog_manager.cc:1347] Generated new cluster ID: 9705f9c721204b82acd692f1bb5958b2
I20260809 07:53:17.333577 30884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:17.361943 30884 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:53:17.363247 30884 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:17.368821 30884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264: Generated new TSK 0
I20260809 07:53:17.369560 30884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
I20260809 07:53:17.400143 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:43097
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:17.505779 30900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:17.506022 30900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:17.506063 30900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:17.509915 30900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:17.509981 30900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:17.510018 30900 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 07:53:17.510052 30900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:53:17.515012 30900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.112.62:43097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:17.515903 30900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:17.516819 30900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:17.525625 30912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:17.525620 30906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:17.525667 30908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:17.525852 30900 server_base.cc:1034] running on GCE node
I20260809 07:53:17.526294 30900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:17.526957 30900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:17.528127 30900 hybrid_clock.cc:648] HybridClock initialized: now 1786261997528106 us; error 33 us; skew 500 ppm
Aug 09 07:53:17 dist-test-slave-vpvm krb5kdc[30730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261997, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:17.530977 30900 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:53:17.531782 30900 webserver.cc:460] Webserver started at http://127.28.112.1:42205/ using document root <none> and password file <none>
I20260809 07:53:17.532302 30900 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:17.532367 30900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:17.532562 30900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:17.534557 30900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "6d3a4750a5f94828b0e1831992c6e140"
format_stamp: "Formatted at 2026-08-09 07:53:17 on dist-test-slave-vpvm"
server_key: "34303b36f26c41df4b9b08b976a77858"
server_key_iv: "d5e340954160e0b1b5dad1d480e09159"
server_key_version: "encryptionkey@0"
I20260809 07:53:17.535075 30900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "6d3a4750a5f94828b0e1831992c6e140"
format_stamp: "Formatted at 2026-08-09 07:53:17 on dist-test-slave-vpvm"
server_key: "34303b36f26c41df4b9b08b976a77858"
server_key_iv: "d5e340954160e0b1b5dad1d480e09159"
server_key_version: "encryptionkey@0"
I20260809 07:53:17.538551 30900 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 07:53:17.540742 30920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:17.541822 30900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 07:53:17.541929 30900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "6d3a4750a5f94828b0e1831992c6e140"
format_stamp: "Formatted at 2026-08-09 07:53:17 on dist-test-slave-vpvm"
server_key: "34303b36f26c41df4b9b08b976a77858"
server_key_iv: "d5e340954160e0b1b5dad1d480e09159"
server_key_version: "encryptionkey@0"
I20260809 07:53:17.542066 30900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:17.556775 30900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:17.559392 30900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:17.559537 30900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:17.560842 30900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:17.560889 30900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:17.560930 30900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:17.560959 30900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 07:53:17 dist-test-slave-vpvm krb5kdc[30730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261997, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:17.571111 30900 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:41629
I20260809 07:53:17.571141 31101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:41629 every 8 connection(s)
I20260809 07:53:17.572227 30900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:53:17.573802 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:17.561524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:56121 (local address 127.28.112.62:43097)
0809 07:53:17.561680 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:17.561685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:17.562908 (+ 1223us) server_negotiation.cc:407] Connection header received
0809 07:53:17.563944 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:17.563948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:17.564010 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:17.564107 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:17.565244 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.565773 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.567422 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.567778 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.567887 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:17.570251 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:17.570255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:17.570258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:17.570289 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:17.571872 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:17.572346 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:17.572350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:17.572352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:17.572441 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:17.572823 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:17.572826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:17.572828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:17.573055 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:17.573134 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:17.573583 (+ 449us) server_negotiation.cc:299] Negotiation successful
0809 07:53:17.573685 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 07:53:17.574524 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:17.562315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43097 (local address 127.28.112.1:56121)
0809 07:53:17.562758 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:17.562805 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:17.563749 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:17.564287 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:17.564293 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:17.564636 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:17.565079 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:17.565095 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.565880 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.565884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:17.567289 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:17.567301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.567962 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.567967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:17.568127 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:17.568611 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:17.568626 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:17.570088 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:17.571979 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:17.571984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:17.571994 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:17.572229 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:17.572528 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:17.572531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:17.572532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:17.572706 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:17.573155 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:17.573162 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:17.573380 (+ 218us) client_negotiation.cc:769] Sending connection context
0809 07:53:17.573527 (+ 147us) client_negotiation.cc:240] Negotiation successful
0809 07:53:17.573767 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":110,"threads_started":1}
I20260809 07:53:17.575459 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 30900
I20260809 07:53:17.575572 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:53:17.575731 31103 heartbeater.cc:347] Connected to a master server at 127.28.112.62:43097
I20260809 07:53:17.575847 29120 external_mini_cluster.cc:1428] Setting key 1e1a111cd8466bf561b122935c8d5272
I20260809 07:53:17.575906 31103 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:17.576469 31103 heartbeater.cc:511] Master 127.28.112.62:43097 requested a full tablet report, sending...
I20260809 07:53:17.578115 30797 ts_manager.cc:194] Registered new tserver with Master: 6d3a4750a5f94828b0e1831992c6e140 (127.28.112.1:41629)
I20260809 07:53:17.579502 30797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:56121
I20260809 07:53:17.586004 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:17.579487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:38957 (local address 127.28.112.62:43097)
0809 07:53:17.579622 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:17.579627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:17.579641 (+ 14us) server_negotiation.cc:407] Connection header received
0809 07:53:17.579675 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:17.579678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:17.579718 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:17.579790 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:17.580270 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.580917 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.581919 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.582168 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.582241 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:17.583340 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:17.583343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:17.583345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:17.583367 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:17.584569 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:17.585004 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:17.585007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:17.585008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:17.585071 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:17.585312 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:17.585314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:17.585315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:17.585456 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:17.585524 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:17.585786 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 07:53:17.585869 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 07:53:17.586091 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:17.578239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43097 (local address 127.28.112.1:38957)
0809 07:53:17.578487 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:17.578502 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:17.578584 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:17.579836 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:17.579839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:17.580032 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:17.580160 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:17.580168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.581047 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.581051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:17.581823 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:17.581833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.582269 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.582273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:17.582360 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:17.582763 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:17.582776 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:17.583189 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:17.584674 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:17.584678 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:17.584680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:17.584920 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:17.585141 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:17.585144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:17.585146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:17.585238 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:17.585531 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:17.585541 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:17.585637 (+ 96us) client_negotiation.cc:769] Sending connection context
0809 07:53:17.585737 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 07:53:17.585866 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.28.112.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
I20260809 07:53:17.601913 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:43097
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:17.707492 31120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:17.707705 31120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:17.707746 31120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:17.711549 31120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:17.711597 31120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:17.711633 31120 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 07:53:17.711663 31120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:53:17.716522 31120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.112.62:43097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:17.717401 31120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:17.718338 31120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:17.727056 31130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:17.727108 31127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:17.727052 31128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:17.727211 31120 server_base.cc:1034] running on GCE node
I20260809 07:53:17.727713 31120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:17.728248 31120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:17.729398 31120 hybrid_clock.cc:648] HybridClock initialized: now 1786261997729373 us; error 36 us; skew 500 ppm
Aug 09 07:53:17 dist-test-slave-vpvm krb5kdc[30730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261997, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:17.731930 31120 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:53:17.732729 31120 webserver.cc:460] Webserver started at http://127.28.112.2:37615/ using document root <none> and password file <none>
I20260809 07:53:17.733206 31120 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:17.733258 31120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:17.733425 31120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:17.735296 31120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "da1833a5e4b74903a53c461c585ee07c"
format_stamp: "Formatted at 2026-08-09 07:53:17 on dist-test-slave-vpvm"
server_key: "b3655a2b93dfb3256e339048c3c428d1"
server_key_iv: "7bbb0f24d8a4c0ecc1193a99c290b143"
server_key_version: "encryptionkey@0"
I20260809 07:53:17.735773 31120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "da1833a5e4b74903a53c461c585ee07c"
format_stamp: "Formatted at 2026-08-09 07:53:17 on dist-test-slave-vpvm"
server_key: "b3655a2b93dfb3256e339048c3c428d1"
server_key_iv: "7bbb0f24d8a4c0ecc1193a99c290b143"
server_key_version: "encryptionkey@0"
I20260809 07:53:17.739082 31120 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:53:17.741216 31138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:17.742329 31120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:17.742421 31120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "da1833a5e4b74903a53c461c585ee07c"
format_stamp: "Formatted at 2026-08-09 07:53:17 on dist-test-slave-vpvm"
server_key: "b3655a2b93dfb3256e339048c3c428d1"
server_key_iv: "7bbb0f24d8a4c0ecc1193a99c290b143"
server_key_version: "encryptionkey@0"
I20260809 07:53:17.742494 31120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:17.757081 31120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:17.759811 31120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:17.759948 31120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:17.761237 31120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:17.761289 31120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:17.761340 31120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:17.761366 31120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 07:53:17 dist-test-slave-vpvm krb5kdc[30730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261997, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:17.771392 31120 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:32931
I20260809 07:53:17.771461 31306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:32931 every 8 connection(s)
I20260809 07:53:17.772377 31120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:53:17.773561 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:17.761882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:58283 (local address 127.28.112.62:43097)
0809 07:53:17.762042 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:17.762047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:17.762894 (+ 847us) server_negotiation.cc:407] Connection header received
0809 07:53:17.763856 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:17.763860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:17.763917 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:17.763991 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:17.765141 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.765931 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.767186 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.767479 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.767577 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:17.770166 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:17.770170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:17.770172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:17.770197 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:17.771567 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:17.772110 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:17.772113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:17.772114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:17.772189 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:17.772518 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:17.772520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:17.772521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:17.772699 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:17.772800 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:17.773345 (+ 545us) server_negotiation.cc:299] Negotiation successful
0809 07:53:17.773444 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 07:53:17.774262 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:17.762281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43097 (local address 127.28.112.2:58283)
0809 07:53:17.762729 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:17.762778 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:17.763630 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:17.764172 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:17.764179 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:17.764536 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:17.764985 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:17.765002 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.766060 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.766064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:17.767051 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:17.767062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.767643 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.767648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:17.767797 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:17.768297 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:17.768317 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:17.770012 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:17.771689 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:17.771695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:17.771707 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:17.772012 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:17.772263 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:17.772266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:17.772268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:17.772433 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:17.772812 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:17.772818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:17.773088 (+ 270us) client_negotiation.cc:769] Sending connection context
0809 07:53:17.773282 (+ 194us) client_negotiation.cc:240] Negotiation successful
0809 07:53:17.773509 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":109,"threads_started":1}
I20260809 07:53:17.775362 31307 heartbeater.cc:347] Connected to a master server at 127.28.112.62:43097
I20260809 07:53:17.775497 31307 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:17.775933 31307 heartbeater.cc:511] Master 127.28.112.62:43097 requested a full tablet report, sending...
I20260809 07:53:17.776922 30798 ts_manager.cc:194] Registered new tserver with Master: da1833a5e4b74903a53c461c585ee07c (127.28.112.2:32931)
I20260809 07:53:17.776957 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 31120
I20260809 07:53:17.777057 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:53:17.777324 29120 external_mini_cluster.cc:1428] Setting key 994f7001b9f5990f4419ba62e9ee02fb
I20260809 07:53:17.777510 30798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:58283
I20260809 07:53:17.784535 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:17.778004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:42343 (local address 127.28.112.62:43097)
0809 07:53:17.778102 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:17.778106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:17.778750 (+ 644us) server_negotiation.cc:407] Connection header received
0809 07:53:17.778853 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:17.778856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:17.778895 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:17.778951 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:17.779333 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.779888 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.780642 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.780855 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.780936 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:17.781835 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:17.781838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:17.781841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:17.781864 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:17.783089 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:17.783464 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:17.783467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:17.783469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:17.783542 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:17.783780 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:17.783784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:17.783785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:17.783978 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:17.784070 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:17.784303 (+ 233us) server_negotiation.cc:299] Negotiation successful
0809 07:53:17.784398 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 07:53:17.784557 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:17.778372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43097 (local address 127.28.112.2:42343)
0809 07:53:17.778673 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:17.778684 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:17.778770 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:17.778977 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:17.778980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:17.779148 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:17.779233 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:17.779238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.780000 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.780004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:17.780531 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:17.780547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.780949 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.780953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:17.781039 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:17.781424 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:17.781434 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:17.781715 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:17.783196 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:17.783200 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:17.783202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:17.783379 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:17.783622 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:17.783625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:17.783627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:17.783702 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:17.784070 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:17.784073 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:17.784158 (+ 85us) client_negotiation.cc:769] Sending connection context
0809 07:53:17.784240 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 07:53:17.784365 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.28.112.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
I20260809 07:53:17.802428 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:43097
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:17.910714 31323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:17.910949 31323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:17.910991 31323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:17.914891 31323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:17.914937 31323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:17.914979 31323 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 07:53:17.915016 31323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:53:17.919972 31323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.112.62:43097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:17.920867 31323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:17.921800 31323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:17.930995 31336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:17.931001 31333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:17.931150 31323 server_base.cc:1034] running on GCE node
W20260809 07:53:17.931053 31332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:17.931608 31323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:17.932188 31323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:17.933342 31323 hybrid_clock.cc:648] HybridClock initialized: now 1786261997933324 us; error 26 us; skew 500 ppm
Aug 09 07:53:17 dist-test-slave-vpvm krb5kdc[30730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261997, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:17.935909 31323 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:53:17.936721 31323 webserver.cc:460] Webserver started at http://127.28.112.3:45161/ using document root <none> and password file <none>
I20260809 07:53:17.937213 31323 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:17.937269 31323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:17.937427 31323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:17.939347 31323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "53085041937b4ab7bcf7a9cd243fd73f"
format_stamp: "Formatted at 2026-08-09 07:53:17 on dist-test-slave-vpvm"
server_key: "f94cec0a2919dfcf8b3ebb82666058e5"
server_key_iv: "61d7a806b4144ddfb8c5622ff7bb1eb0"
server_key_version: "encryptionkey@0"
I20260809 07:53:17.939842 31323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "53085041937b4ab7bcf7a9cd243fd73f"
format_stamp: "Formatted at 2026-08-09 07:53:17 on dist-test-slave-vpvm"
server_key: "f94cec0a2919dfcf8b3ebb82666058e5"
server_key_iv: "61d7a806b4144ddfb8c5622ff7bb1eb0"
server_key_version: "encryptionkey@0"
I20260809 07:53:17.943254 31323 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:53:17.945525 31348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:17.946750 31323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:17.946843 31323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "53085041937b4ab7bcf7a9cd243fd73f"
format_stamp: "Formatted at 2026-08-09 07:53:17 on dist-test-slave-vpvm"
server_key: "f94cec0a2919dfcf8b3ebb82666058e5"
server_key_iv: "61d7a806b4144ddfb8c5622ff7bb1eb0"
server_key_version: "encryptionkey@0"
I20260809 07:53:17.946921 31323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:17.961215 31323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:17.963702 31323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:17.963842 31323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:17.965155 31323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:17.965204 31323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:17.965240 31323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:17.965257 31323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 07:53:17 dist-test-slave-vpvm krb5kdc[30730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261997, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:17.975476 31323 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:46055
I20260809 07:53:17.975522 31527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:46055 every 8 connection(s)
I20260809 07:53:17.976392 31323 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:53:17.977550 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 31323
I20260809 07:53:17.977638 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:53:17.977864 29120 external_mini_cluster.cc:1428] Setting key d366c6200333f5e5a11491a84c4a72cf
I20260809 07:53:17.978377 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:17.965748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:33783 (local address 127.28.112.62:43097)
0809 07:53:17.965893 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:17.965898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:17.966750 (+ 852us) server_negotiation.cc:407] Connection header received
0809 07:53:17.967870 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:17.967873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:17.967930 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:17.968015 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:17.969233 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.969737 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.970949 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.971173 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.971261 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:17.974631 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:17.974634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:17.974636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:17.974662 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:17.976199 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:17.976734 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:17.976737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:17.976738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:17.976819 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:17.977211 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:17.977214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:17.977216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:17.977469 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:17.977558 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:17.978090 (+ 532us) server_negotiation.cc:299] Negotiation successful
0809 07:53:17.978217 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 07:53:17.979209 31359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:17.966130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43097 (local address 127.28.112.3:33783)
0809 07:53:17.966575 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:17.966628 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:17.967639 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:17.968302 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:17.968311 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:17.968662 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:17.969087 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:17.969100 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.969855 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.969858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:17.970832 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:17.970842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.971309 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.971313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:17.971473 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:17.971992 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:17.972012 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:17.974468 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:17.976308 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:17.976313 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:17.976326 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:17.976641 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:17.976908 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:17.976912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:17.976914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:17.977090 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:17.977575 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:17.977581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:17.977836 (+ 255us) client_negotiation.cc:769] Sending connection context
0809 07:53:17.978022 (+ 186us) client_negotiation.cc:240] Negotiation successful
0809 07:53:17.978318 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":103,"threads_started":1}
I20260809 07:53:17.980428 31528 heartbeater.cc:347] Connected to a master server at 127.28.112.62:43097
I20260809 07:53:17.980561 31528 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:17.981088 31528 heartbeater.cc:511] Master 127.28.112.62:43097 requested a full tablet report, sending...
I20260809 07:53:17.982142 30798 ts_manager.cc:194] Registered new tserver with Master: 53085041937b4ab7bcf7a9cd243fd73f (127.28.112.3:46055)
I20260809 07:53:17.982856 30798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:33783
I20260809 07:53:17.988860 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:17.982647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:35713 (local address 127.28.112.62:43097)
0809 07:53:17.982748 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:17.982753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:17.982902 (+ 149us) server_negotiation.cc:407] Connection header received
0809 07:53:17.983044 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:17.983047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:17.983092 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:17.983175 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:17.983632 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.984412 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.985220 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.985424 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.985501 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:17.986481 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:17.986483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:17.986485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:17.986503 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:17.987666 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:17.988031 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:17.988033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:17.988040 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:17.988101 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:17.988291 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:17.988292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:17.988294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:17.988432 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:17.988498 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:17.988674 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 07:53:17.988757 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 07:53:17.988879 31538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:17.982593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43097 (local address 127.28.112.3:35713)
0809 07:53:17.982832 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:17.982843 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:17.982930 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:17.983200 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:17.983203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:17.983410 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:17.983520 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:17.983526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.984562 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.984565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:17.985115 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:17.985123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:17.985521 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:17.985524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:17.985597 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:17.986047 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:17.986061 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:17.986363 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:17.987770 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:17.987773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:17.987775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:17.987957 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:17.988162 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:17.988164 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:17.988166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:17.988233 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:17.988492 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:17.988493 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:17.988561 (+ 68us) client_negotiation.cc:769] Sending connection context
0809 07:53:17.988632 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 07:53:17.988741 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20260809 07:53:17.991802 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 07:53:18 dist-test-slave-vpvm krb5kdc[30730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 07:53:18 dist-test-slave-vpvm krb5kdc[30730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:18.015398 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.008514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58370 (local address 127.28.112.62:43097)
0809 07:53:18.008628 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:18.008631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:18.008646 (+ 15us) server_negotiation.cc:407] Connection header received
0809 07:53:18.008683 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:18.008685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:18.008722 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:18.008786 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:18.009397 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.009897 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.010643 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.010845 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.010914 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.012987 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:18.012990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:18.012993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:18.013016 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:18.014190 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:18.014527 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:18.014530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:18.014531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:18.014597 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:18.014772 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:18.014774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:18.014775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:18.014931 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:18.015022 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:18.015197 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 07:53:18.015283 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 07:53:18.018095 30798 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:53:18.020972 30798 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:53:18.034329 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.028894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:46055 (local address 127.0.0.1:58814)
0809 07:53:18.029196 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:18.029239 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:18.029355 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:18.030110 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:18.030116 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:18.030141 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:18.030266 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.030273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.031290 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.031293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.032665 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.032678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.033757 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.033761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.033870 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.033910 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 07:53:18.034059 (+ 149us) client_negotiation.cc:240] Negotiation successful
0809 07:53:18.034182 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":29,"thread_start_us":72,"threads_started":1}
I20260809 07:53:18.034549 31564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.029898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:41629 (local address 127.0.0.1:52792)
0809 07:53:18.030223 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:18.030236 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:18.030324 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:18.031110 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:18.031113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:18.031141 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:18.031241 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.031247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.032260 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.032263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.033202 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.033212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.034142 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.034145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.034239 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.034262 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 07:53:18.034346 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 07:53:18.034410 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":4,"thread_start_us":137,"threads_started":1}
I20260809 07:53:18.034857 31565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.030367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.28.112.1:41629)
0809 07:53:18.030682 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:18.030693 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:18.030714 (+ 21us) server_negotiation.cc:407] Connection header received
0809 07:53:18.030787 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:18.030792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:18.030967 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:18.031123 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:18.031352 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.032128 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.033326 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.034006 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.034135 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.034283 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:18.034380 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 07:53:18.034714 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":110,"threads_started":1}
I20260809 07:53:18.035405 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.029058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.28.112.3:46055)
0809 07:53:18.029393 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:18.029406 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:18.029431 (+ 25us) server_negotiation.cc:407] Connection header received
0809 07:53:18.029510 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:18.029515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:18.029738 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:18.029905 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:18.030387 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.031165 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.032812 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.033610 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.033729 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.033892 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:18.035201 (+ 1309us) server_negotiation.cc:299] Negotiation successful
0809 07:53:18.035277 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":130,"threads_started":1}
I20260809 07:53:18.035998 31562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.029576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:32931 (local address 127.0.0.1:43086)
0809 07:53:18.030183 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:18.030196 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:18.030277 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:18.031271 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:18.031274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:18.031291 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:18.031445 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.031452 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.032811 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.032814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.033926 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.033937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.035588 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.035591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.035704 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.035728 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 07:53:18.035803 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 07:53:18.035868 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":52,"thread_start_us":96,"threads_started":1}
I20260809 07:53:18.036074 31563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.029763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.28.112.2:32931)
0809 07:53:18.030779 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:18.030789 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:18.030807 (+ 18us) server_negotiation.cc:407] Connection header received
0809 07:53:18.030869 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:18.030873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:18.031011 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:18.031145 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:18.031574 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.032683 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.034069 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.034880 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.034990 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.035135 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:18.035841 (+ 706us) server_negotiation.cc:299] Negotiation successful
0809 07:53:18.035933 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925,"thread_start_us":112,"threads_started":1}
I20260809 07:53:18.037880 31003 tablet_service.cc:1467] Processing CreateTablet for tablet c0015140dd674b41bab9cf5ae6b06cae (DEFAULT_TABLE table=test-table [id=08eebcb436d749a08ddd35227709954d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:18.037881 31433 tablet_service.cc:1467] Processing CreateTablet for tablet c0015140dd674b41bab9cf5ae6b06cae (DEFAULT_TABLE table=test-table [id=08eebcb436d749a08ddd35227709954d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:18.038843 31433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0015140dd674b41bab9cf5ae6b06cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:18.038883 31003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0015140dd674b41bab9cf5ae6b06cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:18.041075 31212 tablet_service.cc:1467] Processing CreateTablet for tablet c0015140dd674b41bab9cf5ae6b06cae (DEFAULT_TABLE table=test-table [id=08eebcb436d749a08ddd35227709954d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:18.042207 31212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0015140dd674b41bab9cf5ae6b06cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:18.045593 31566 tablet_bootstrap.cc:492] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140: Bootstrap starting.
I20260809 07:53:18.048041 31566 tablet_bootstrap.cc:654] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:18.048751 31566 log.cc:824] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:18.049350 31567 tablet_bootstrap.cc:492] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f: Bootstrap starting.
I20260809 07:53:18.049901 31571 tablet_bootstrap.cc:492] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c: Bootstrap starting.
I20260809 07:53:18.050779 31566 tablet_bootstrap.cc:492] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140: No bootstrap required, opened a new log
I20260809 07:53:18.050961 31566 ts_tablet_manager.cc:1397] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 07:53:18.051432 31567 tablet_bootstrap.cc:654] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:18.052076 31571 tablet_bootstrap.cc:654] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:18.052251 31567 log.cc:824] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:18.052855 31571 log.cc:824] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:18.054170 31567 tablet_bootstrap.cc:492] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f: No bootstrap required, opened a new log
I20260809 07:53:18.054352 31567 ts_tablet_manager.cc:1397] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 07:53:18.054431 31566 raft_consensus.cc:348] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } }
I20260809 07:53:18.054713 31566 raft_consensus.cc:374] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:18.054807 31566 raft_consensus.cc:729] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d3a4750a5f94828b0e1831992c6e140, State: Initialized, Role: FOLLOWER
I20260809 07:53:18.054981 31571 tablet_bootstrap.cc:492] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c: No bootstrap required, opened a new log
I20260809 07:53:18.055187 31571 ts_tablet_manager.cc:1397] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260809 07:53:18.055255 31566 consensus_queue.cc:260] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } }
I20260809 07:53:18.055938 31103 heartbeater.cc:503] Master 127.28.112.62:43097 was elected leader, sending a full tablet report...
I20260809 07:53:18.056231 31566 ts_tablet_manager.cc:1428] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 07:53:18.058152 31571 raft_consensus.cc:348] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } }
I20260809 07:53:18.058344 31571 raft_consensus.cc:374] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:18.058430 31571 raft_consensus.cc:729] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da1833a5e4b74903a53c461c585ee07c, State: Initialized, Role: FOLLOWER
I20260809 07:53:18.058845 31571 consensus_queue.cc:260] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } }
I20260809 07:53:18.058923 31567 raft_consensus.cc:348] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } }
I20260809 07:53:18.059157 31567 raft_consensus.cc:374] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:18.059255 31567 raft_consensus.cc:729] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53085041937b4ab7bcf7a9cd243fd73f, State: Initialized, Role: FOLLOWER
I20260809 07:53:18.059521 31307 heartbeater.cc:503] Master 127.28.112.62:43097 was elected leader, sending a full tablet report...
I20260809 07:53:18.059715 31567 consensus_queue.cc:260] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } }
I20260809 07:53:18.060613 31567 ts_tablet_manager.cc:1428] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 07:53:18.060611 31571 ts_tablet_manager.cc:1428] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260809 07:53:18.060850 31528 heartbeater.cc:503] Master 127.28.112.62:43097 was elected leader, sending a full tablet report...
W20260809 07:53:18.073371 31108 tablet.cc:2369] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:53:18.078805 31578 raft_consensus.cc:482] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:18.078945 31578 raft_consensus.cc:504] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } }
I20260809 07:53:18.079981 31578 leader_election.cc:288] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da1833a5e4b74903a53c461c585ee07c (127.28.112.2:32931), 6d3a4750a5f94828b0e1831992c6e140 (127.28.112.1:41629)
I20260809 07:53:18.084393 31359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.080180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:32931 (local address 127.28.112.3:57023)
0809 07:53:18.080317 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:18.080337 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:18.080446 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:18.080842 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:18.080846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:18.080860 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:18.080977 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.080982 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.082035 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.082039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.083327 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.083339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.084050 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.084053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.084135 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.084155 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 07:53:18.084211 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 07:53:18.084269 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260809 07:53:18.084430 31563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.080281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:57023 (local address 127.28.112.2:32931)
0809 07:53:18.080413 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:18.080418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:18.080431 (+ 13us) server_negotiation.cc:407] Connection header received
0809 07:53:18.080693 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:18.080697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:18.080744 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:18.080821 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:18.081078 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.081890 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.083446 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.083943 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.084057 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.084108 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:18.084241 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 07:53:18.084313 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 07:53:18.084753 31565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.080580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:59275 (local address 127.28.112.1:41629)
0809 07:53:18.080708 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:18.080712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:18.080801 (+ 89us) server_negotiation.cc:407] Connection header received
0809 07:53:18.080906 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:18.080909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:18.080949 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:18.081003 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:18.081237 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.081968 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.083689 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.084232 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.084341 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.084384 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:18.084585 (+ 201us) server_negotiation.cc:299] Negotiation successful
0809 07:53:18.084654 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 07:53:18.084983 31241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0015140dd674b41bab9cf5ae6b06cae" candidate_uuid: "53085041937b4ab7bcf7a9cd243fd73f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da1833a5e4b74903a53c461c585ee07c" is_pre_election: true
I20260809 07:53:18.085093 31579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.080429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:41629 (local address 127.28.112.3:59275)
0809 07:53:18.080724 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:18.080735 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:18.080826 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:18.081025 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:18.081028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:18.081054 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:18.081150 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.081155 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.082050 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.082052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.083585 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.083592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.084347 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.084351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.084437 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.084460 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 07:53:18.084882 (+ 422us) client_negotiation.cc:240] Negotiation successful
0809 07:53:18.084953 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":11,"thread_start_us":117,"threads_started":1}
I20260809 07:53:18.085259 31241 raft_consensus.cc:2454] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53085041937b4ab7bcf7a9cd243fd73f in term 0.
I20260809 07:53:18.085660 31349 leader_election.cc:302] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53085041937b4ab7bcf7a9cd243fd73f, da1833a5e4b74903a53c461c585ee07c; no voters:
I20260809 07:53:18.085889 31029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0015140dd674b41bab9cf5ae6b06cae" candidate_uuid: "53085041937b4ab7bcf7a9cd243fd73f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d3a4750a5f94828b0e1831992c6e140" is_pre_election: true
I20260809 07:53:18.085919 31578 raft_consensus.cc:2781] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:53:18.086026 31578 raft_consensus.cc:482] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:53:18.086081 31578 raft_consensus.cc:3037] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:18.086170 31029 raft_consensus.cc:2454] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53085041937b4ab7bcf7a9cd243fd73f in term 0.
I20260809 07:53:18.087167 31578 raft_consensus.cc:504] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } }
I20260809 07:53:18.087466 31578 leader_election.cc:288] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [CANDIDATE]: Term 1 election: Requested vote from peers da1833a5e4b74903a53c461c585ee07c (127.28.112.2:32931), 6d3a4750a5f94828b0e1831992c6e140 (127.28.112.1:41629)
I20260809 07:53:18.087778 31241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0015140dd674b41bab9cf5ae6b06cae" candidate_uuid: "53085041937b4ab7bcf7a9cd243fd73f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da1833a5e4b74903a53c461c585ee07c"
I20260809 07:53:18.087821 31029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0015140dd674b41bab9cf5ae6b06cae" candidate_uuid: "53085041937b4ab7bcf7a9cd243fd73f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d3a4750a5f94828b0e1831992c6e140"
I20260809 07:53:18.087886 31241 raft_consensus.cc:3037] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:18.087914 31029 raft_consensus.cc:3037] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:18.088833 31029 raft_consensus.cc:2454] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53085041937b4ab7bcf7a9cd243fd73f in term 1.
I20260809 07:53:18.089107 31349 leader_election.cc:302] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53085041937b4ab7bcf7a9cd243fd73f, 6d3a4750a5f94828b0e1831992c6e140; no voters:
I20260809 07:53:18.089406 31241 raft_consensus.cc:2454] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53085041937b4ab7bcf7a9cd243fd73f in term 1.
I20260809 07:53:18.089605 31578 raft_consensus.cc:2781] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:18.089797 31578 raft_consensus.cc:686] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [term 1 LEADER]: Becoming Leader. State: Replica: 53085041937b4ab7bcf7a9cd243fd73f, State: Running, Role: LEADER
I20260809 07:53:18.090113 31578 consensus_queue.cc:237] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } }
I20260809 07:53:18.093178 30797 catalog_manager.cc:5495] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f reported cstate change: term changed from 0 to 1, leader changed from <none> to 53085041937b4ab7bcf7a9cd243fd73f (127.28.112.3). New cstate: current_term: 1 leader_uuid: "53085041937b4ab7bcf7a9cd243fd73f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } health_report { overall_health: UNKNOWN } } }
I20260809 07:53:18.118049 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.114950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58820 (local address 127.28.112.3:46055)
0809 07:53:18.115056 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:18.115059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:18.115097 (+ 38us) server_negotiation.cc:407] Connection header received
0809 07:53:18.115280 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:18.115283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:18.115336 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:18.115415 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:18.115591 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.116075 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.116845 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.117047 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.117112 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.117354 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:18.117727 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:18.117806 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:18.117861 (+ 55us) server_negotiation.cc:299] Negotiation successful
0809 07:53:18.117907 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 07:53:18.123484 31029 raft_consensus.cc:1261] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140 [term 1 FOLLOWER]: Refusing update from remote peer 53085041937b4ab7bcf7a9cd243fd73f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:18.124135 31578 consensus_queue.cc:1035] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:18.125545 31241 raft_consensus.cc:1261] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c [term 1 FOLLOWER]: Refusing update from remote peer 53085041937b4ab7bcf7a9cd243fd73f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:18.126121 31580 consensus_queue.cc:1035] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [LEADER]: Connected to new peer: Peer: permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:18.131325 31587 mvcc.cc:205] Tried to move back new op lower bound from 7316529144306647040 to 7316529144179978240. Current Snapshot: MvccSnapshot[applied={T|T < 7316529144306647040}]
I20260809 07:53:18.132793 31586 mvcc.cc:205] Tried to move back new op lower bound from 7316529144306647040 to 7316529144179978240. Current Snapshot: MvccSnapshot[applied={T|T < 7316529144306647040}]
I20260809 07:53:18.134510 31590 mvcc.cc:205] Tried to move back new op lower bound from 7316529144306647040 to 7316529144179978240. Current Snapshot: MvccSnapshot[applied={T|T < 7316529144306647040}]
I20260809 07:53:18.143270 30798 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:18.143488 30798 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:18.146075 30798 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264: Sending DeleteTablet for 3 replicas of tablet c0015140dd674b41bab9cf5ae6b06cae
I20260809 07:53:18.147019 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet c0015140dd674b41bab9cf5ae6b06cae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:18 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:58814
I20260809 07:53:18.147128 31212 tablet_service.cc:1514] Processing DeleteTablet for tablet c0015140dd674b41bab9cf5ae6b06cae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:18 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:43086
I20260809 07:53:18.147562 30798 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 07:53:18.147776 31607 tablet_replica.cc:330] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f: stopping tablet replica
I20260809 07:53:18.147778 31003 tablet_service.cc:1514] Processing DeleteTablet for tablet c0015140dd674b41bab9cf5ae6b06cae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:18 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:52792
W20260809 07:53:18.147943 30798 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:53:18.148058 31607 raft_consensus.cc:2229] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:53:18.148379 31607 raft_consensus.cc:2258] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:18.148516 31609 tablet_replica.cc:330] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140: stopping tablet replica
I20260809 07:53:18.148846 31609 raft_consensus.cc:2229] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:18.149209 31609 raft_consensus.cc:2258] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:18.149993 31607 ts_tablet_manager.cc:1905] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:18.150815 31609 ts_tablet_manager.cc:1905] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:18.152670 31607 ts_tablet_manager.cc:1918] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:53:18.152781 31607 log.cc:1196] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/c0015140dd674b41bab9cf5ae6b06cae
I20260809 07:53:18.153091 31607 ts_tablet_manager.cc:1939] T c0015140dd674b41bab9cf5ae6b06cae P 53085041937b4ab7bcf7a9cd243fd73f: Deleting consensus metadata
I20260809 07:53:18.153239 31003 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0027e2d96443969465a914edb854ca (DEFAULT_TABLE table=test-table [id=77f2e56192f04b7881a2735d92d9f772]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:18.153493 31003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0027e2d96443969465a914edb854ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:18.153820 31609 ts_tablet_manager.cc:1918] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:53:18.153996 31609 log.cc:1196] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-0/wal/wals/c0015140dd674b41bab9cf5ae6b06cae
I20260809 07:53:18.154357 31609 ts_tablet_manager.cc:1939] T c0015140dd674b41bab9cf5ae6b06cae P 6d3a4750a5f94828b0e1831992c6e140: Deleting consensus metadata
I20260809 07:53:18.155578 31566 tablet_bootstrap.cc:492] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140: Bootstrap starting.
I20260809 07:53:18.155799 31212 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0027e2d96443969465a914edb854ca (DEFAULT_TABLE table=test-table [id=77f2e56192f04b7881a2735d92d9f772]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:18.156049 31212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0027e2d96443969465a914edb854ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:18.156478 31566 tablet_bootstrap.cc:654] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:18.157943 31566 tablet_bootstrap.cc:492] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140: No bootstrap required, opened a new log
I20260809 07:53:18.158035 31571 tablet_bootstrap.cc:492] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c: Bootstrap starting.
I20260809 07:53:18.158051 31566 ts_tablet_manager.cc:1397] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 07:53:18.158418 31433 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0027e2d96443969465a914edb854ca (DEFAULT_TABLE table=test-table [id=77f2e56192f04b7881a2735d92d9f772]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:18.158457 31566 raft_consensus.cc:348] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } }
I20260809 07:53:18.158571 31566 raft_consensus.cc:374] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:18.158601 31566 raft_consensus.cc:729] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d3a4750a5f94828b0e1831992c6e140, State: Initialized, Role: FOLLOWER
I20260809 07:53:18.158665 31433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0027e2d96443969465a914edb854ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:18.158711 31566 consensus_queue.cc:260] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } }
I20260809 07:53:18.158972 31571 tablet_bootstrap.cc:654] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:18.160188 31566 ts_tablet_manager.cc:1428] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:18.160674 31567 tablet_bootstrap.cc:492] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f: Bootstrap starting.
I20260809 07:53:18.161072 30777 catalog_manager.cc:4841] TS 6d3a4750a5f94828b0e1831992c6e140 (127.28.112.1:41629): tablet c0015140dd674b41bab9cf5ae6b06cae (table test-table [id=08eebcb436d749a08ddd35227709954d]) successfully deleted
I20260809 07:53:18.161212 30777 catalog_manager.cc:4841] TS 53085041937b4ab7bcf7a9cd243fd73f (127.28.112.3:46055): tablet c0015140dd674b41bab9cf5ae6b06cae (table test-table [id=08eebcb436d749a08ddd35227709954d]) successfully deleted
I20260809 07:53:18.161527 31567 tablet_bootstrap.cc:654] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:18.161654 31608 tablet_replica.cc:330] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c: stopping tablet replica
I20260809 07:53:18.162137 31608 raft_consensus.cc:2229] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:18.162781 31608 raft_consensus.cc:2258] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:18.163403 31567 tablet_bootstrap.cc:492] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f: No bootstrap required, opened a new log
I20260809 07:53:18.163488 31567 ts_tablet_manager.cc:1397] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 07:53:18.163923 31567 raft_consensus.cc:348] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } }
I20260809 07:53:18.164021 31567 raft_consensus.cc:374] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:18.164052 31567 raft_consensus.cc:729] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53085041937b4ab7bcf7a9cd243fd73f, State: Initialized, Role: FOLLOWER
I20260809 07:53:18.164173 31567 consensus_queue.cc:260] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } }
I20260809 07:53:18.164310 31608 ts_tablet_manager.cc:1905] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:18.164438 31567 ts_tablet_manager.cc:1428] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:18.166507 31608 ts_tablet_manager.cc:1918] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:53:18.166597 31608 log.cc:1196] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/c0015140dd674b41bab9cf5ae6b06cae
I20260809 07:53:18.166625 31571 tablet_bootstrap.cc:492] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c: No bootstrap required, opened a new log
I20260809 07:53:18.166693 31571 ts_tablet_manager.cc:1397] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260809 07:53:18.166903 31608 ts_tablet_manager.cc:1939] T c0015140dd674b41bab9cf5ae6b06cae P da1833a5e4b74903a53c461c585ee07c: Deleting consensus metadata
I20260809 07:53:18.167104 31571 raft_consensus.cc:348] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } }
I20260809 07:53:18.167213 31571 raft_consensus.cc:374] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:18.167243 31571 raft_consensus.cc:729] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da1833a5e4b74903a53c461c585ee07c, State: Initialized, Role: FOLLOWER
I20260809 07:53:18.167358 31571 consensus_queue.cc:260] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } }
I20260809 07:53:18.167579 31571 ts_tablet_manager.cc:1428] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:18.167956 30777 catalog_manager.cc:4841] TS da1833a5e4b74903a53c461c585ee07c (127.28.112.2:32931): tablet c0015140dd674b41bab9cf5ae6b06cae (table test-table [id=08eebcb436d749a08ddd35227709954d]) successfully deleted
W20260809 07:53:18.227505 31529 tablet.cc:2369] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 07:53:18.273703 31308 tablet.cc:2369] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:53:18.417181 31576 raft_consensus.cc:482] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:18.417344 31576 raft_consensus.cc:504] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } }
I20260809 07:53:18.418431 31576 leader_election.cc:288] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d3a4750a5f94828b0e1831992c6e140 (127.28.112.1:41629), 53085041937b4ab7bcf7a9cd243fd73f (127.28.112.3:46055)
I20260809 07:53:18.423234 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.419173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:59739 (local address 127.28.112.3:46055)
0809 07:53:18.419312 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:18.419316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:18.419349 (+ 33us) server_negotiation.cc:407] Connection header received
0809 07:53:18.419489 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:18.419492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:18.419532 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:18.419593 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:18.419806 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.420498 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.421938 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.422561 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.422686 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.422738 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:18.423025 (+ 287us) server_negotiation.cc:299] Negotiation successful
0809 07:53:18.423091 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 07:53:18.423234 31565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.418783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:58495 (local address 127.28.112.1:41629)
0809 07:53:18.418919 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:18.418922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:18.419000 (+ 78us) server_negotiation.cc:407] Connection header received
0809 07:53:18.419221 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:18.419225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:18.419281 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:18.419372 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:18.419637 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.420407 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.421940 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.422561 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.422683 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.422738 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:18.423025 (+ 287us) server_negotiation.cc:299] Negotiation successful
0809 07:53:18.423091 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 07:53:18.423277 31615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.418682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:41629 (local address 127.28.112.2:58495)
0809 07:53:18.418919 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:18.418935 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:18.419047 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:18.419393 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:18.419396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:18.419410 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:18.419535 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.419541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.420509 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.420512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.421823 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.421835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.422670 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.422674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.422863 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.422928 (+ 65us) client_negotiation.cc:769] Sending connection context
0809 07:53:18.423004 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 07:53:18.423086 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":48,"thread_start_us":69,"threads_started":1}
I20260809 07:53:18.423277 31616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.419064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:46055 (local address 127.28.112.2:59739)
0809 07:53:18.419295 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:18.419305 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:18.419386 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:18.419620 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:18.419623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:18.419642 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:18.419718 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.419723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.420582 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.420584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.421823 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.421835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.422670 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.422674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.422862 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.422931 (+ 69us) client_negotiation.cc:769] Sending connection context
0809 07:53:18.423004 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 07:53:18.423086 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":117,"thread_start_us":84,"threads_started":1}
I20260809 07:53:18.424044 31029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0027e2d96443969465a914edb854ca" candidate_uuid: "da1833a5e4b74903a53c461c585ee07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d3a4750a5f94828b0e1831992c6e140" is_pre_election: true
I20260809 07:53:18.424196 31029 raft_consensus.cc:2454] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da1833a5e4b74903a53c461c585ee07c in term 0.
I20260809 07:53:18.424144 31465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0027e2d96443969465a914edb854ca" candidate_uuid: "da1833a5e4b74903a53c461c585ee07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53085041937b4ab7bcf7a9cd243fd73f" is_pre_election: true
I20260809 07:53:18.424376 31465 raft_consensus.cc:2454] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da1833a5e4b74903a53c461c585ee07c in term 0.
I20260809 07:53:18.424566 31143 leader_election.cc:302] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d3a4750a5f94828b0e1831992c6e140, da1833a5e4b74903a53c461c585ee07c; no voters:
I20260809 07:53:18.424842 31576 raft_consensus.cc:2781] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:53:18.424899 31576 raft_consensus.cc:482] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:53:18.424947 31576 raft_consensus.cc:3037] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:18.425930 31576 raft_consensus.cc:504] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } }
I20260809 07:53:18.426277 31576 leader_election.cc:288] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [CANDIDATE]: Term 1 election: Requested vote from peers 6d3a4750a5f94828b0e1831992c6e140 (127.28.112.1:41629), 53085041937b4ab7bcf7a9cd243fd73f (127.28.112.3:46055)
I20260809 07:53:18.426643 31029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0027e2d96443969465a914edb854ca" candidate_uuid: "da1833a5e4b74903a53c461c585ee07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d3a4750a5f94828b0e1831992c6e140"
I20260809 07:53:18.426697 31465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0027e2d96443969465a914edb854ca" candidate_uuid: "da1833a5e4b74903a53c461c585ee07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53085041937b4ab7bcf7a9cd243fd73f"
I20260809 07:53:18.426755 31029 raft_consensus.cc:3037] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:18.426797 31465 raft_consensus.cc:3037] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:18.427621 31029 raft_consensus.cc:2454] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da1833a5e4b74903a53c461c585ee07c in term 1.
I20260809 07:53:18.427809 31465 raft_consensus.cc:2454] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da1833a5e4b74903a53c461c585ee07c in term 1.
I20260809 07:53:18.428062 31143 leader_election.cc:302] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d3a4750a5f94828b0e1831992c6e140, da1833a5e4b74903a53c461c585ee07c; no voters:
I20260809 07:53:18.428216 31576 raft_consensus.cc:2781] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:18.428395 31576 raft_consensus.cc:686] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [term 1 LEADER]: Becoming Leader. State: Replica: da1833a5e4b74903a53c461c585ee07c, State: Running, Role: LEADER
I20260809 07:53:18.428679 31576 consensus_queue.cc:237] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } }
I20260809 07:53:18.430765 30798 catalog_manager.cc:5495] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c reported cstate change: term changed from 0 to 1, leader changed from <none> to da1833a5e4b74903a53c461c585ee07c (127.28.112.2). New cstate: current_term: 1 leader_uuid: "da1833a5e4b74903a53c461c585ee07c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } health_report { overall_health: UNKNOWN } } }
Aug 09 07:53:18 dist-test-slave-vpvm krb5kdc[30730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261997, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:18.502882 31628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.495609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43097 (local address 127.0.0.1:58374)
0809 07:53:18.495869 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:18.495881 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:18.495984 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:18.496195 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:18.496197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:18.496594 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:18.496690 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.496696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.497344 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.497346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.497889 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.497895 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.498286 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.498288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.498373 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.498893 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:18.498905 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:18.500232 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:18.501554 (+ 1322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:18.501562 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:18.501568 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:18.501782 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:18.502026 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:18.502028 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:18.502029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:18.502111 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:18.502413 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:18.502418 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:18.502514 (+ 96us) client_negotiation.cc:769] Sending connection context
0809 07:53:18.502608 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 07:53:18.502722 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":14,"thread_start_us":88,"threads_started":1}
I20260809 07:53:18.502897 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.495673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58374 (local address 127.28.112.62:43097)
0809 07:53:18.495823 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:18.495827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:18.495947 (+ 120us) server_negotiation.cc:407] Connection header received
0809 07:53:18.496055 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:18.496057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:18.496099 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:18.496169 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:18.496777 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.497272 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.498003 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.498207 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.498277 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.500360 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:18.500364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:18.500366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:18.500388 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:18.501450 (+ 1062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:18.501854 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:18.501856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:18.501857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:18.501920 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:18.502178 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:18.502180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:18.502181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:18.502339 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:18.502411 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:18.502658 (+ 247us) server_negotiation.cc:299] Negotiation successful
0809 07:53:18.502773 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":63}
I20260809 07:53:18.506664 30798 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:58374:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 07:53:18.507339 30798 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:53:18.511852 31003 tablet_service.cc:1467] Processing CreateTablet for tablet 29a93f8bcaf64e7eb48b4abf44e43d4b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d111745b93734b928489e02a20795c8a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 07:53:18.511901 31212 tablet_service.cc:1467] Processing CreateTablet for tablet 29a93f8bcaf64e7eb48b4abf44e43d4b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d111745b93734b928489e02a20795c8a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 07:53:18.511934 31433 tablet_service.cc:1467] Processing CreateTablet for tablet 29a93f8bcaf64e7eb48b4abf44e43d4b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d111745b93734b928489e02a20795c8a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 07:53:18.512283 31212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29a93f8bcaf64e7eb48b4abf44e43d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:18.512283 31433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29a93f8bcaf64e7eb48b4abf44e43d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:18.512284 31003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29a93f8bcaf64e7eb48b4abf44e43d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:18.515413 31571 tablet_bootstrap.cc:492] T 29a93f8bcaf64e7eb48b4abf44e43d4b P da1833a5e4b74903a53c461c585ee07c: Bootstrap starting.
I20260809 07:53:18.515470 31566 tablet_bootstrap.cc:492] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 6d3a4750a5f94828b0e1831992c6e140: Bootstrap starting.
I20260809 07:53:18.516327 31566 tablet_bootstrap.cc:654] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 6d3a4750a5f94828b0e1831992c6e140: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:18.516422 31571 tablet_bootstrap.cc:654] T 29a93f8bcaf64e7eb48b4abf44e43d4b P da1833a5e4b74903a53c461c585ee07c: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:18.517658 31567 tablet_bootstrap.cc:492] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f: Bootstrap starting.
I20260809 07:53:18.517737 31571 tablet_bootstrap.cc:492] T 29a93f8bcaf64e7eb48b4abf44e43d4b P da1833a5e4b74903a53c461c585ee07c: No bootstrap required, opened a new log
I20260809 07:53:18.517813 31571 ts_tablet_manager.cc:1397] T 29a93f8bcaf64e7eb48b4abf44e43d4b P da1833a5e4b74903a53c461c585ee07c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:18.518270 31571 raft_consensus.cc:348] T 29a93f8bcaf64e7eb48b4abf44e43d4b P da1833a5e4b74903a53c461c585ee07c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } }
I20260809 07:53:18.518378 31571 raft_consensus.cc:374] T 29a93f8bcaf64e7eb48b4abf44e43d4b P da1833a5e4b74903a53c461c585ee07c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:18.518409 31571 raft_consensus.cc:729] T 29a93f8bcaf64e7eb48b4abf44e43d4b P da1833a5e4b74903a53c461c585ee07c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da1833a5e4b74903a53c461c585ee07c, State: Initialized, Role: FOLLOWER
I20260809 07:53:18.518529 31571 consensus_queue.cc:260] T 29a93f8bcaf64e7eb48b4abf44e43d4b P da1833a5e4b74903a53c461c585ee07c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } }
I20260809 07:53:18.518664 31567 tablet_bootstrap.cc:654] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:18.518903 31576 tablet_replica.cc:439] T 29a93f8bcaf64e7eb48b4abf44e43d4b P da1833a5e4b74903a53c461c585ee07c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } } }
I20260809 07:53:18.519029 31576 tablet_replica.cc:442] T 29a93f8bcaf64e7eb48b4abf44e43d4b P da1833a5e4b74903a53c461c585ee07c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:18.519635 31571 ts_tablet_manager.cc:1428] T 29a93f8bcaf64e7eb48b4abf44e43d4b P da1833a5e4b74903a53c461c585ee07c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:18.519697 31566 tablet_bootstrap.cc:492] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 6d3a4750a5f94828b0e1831992c6e140: No bootstrap required, opened a new log
I20260809 07:53:18.519778 31566 ts_tablet_manager.cc:1397] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 6d3a4750a5f94828b0e1831992c6e140: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260809 07:53:18.520148 31566 raft_consensus.cc:348] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 6d3a4750a5f94828b0e1831992c6e140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } }
I20260809 07:53:18.520231 31566 raft_consensus.cc:374] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 6d3a4750a5f94828b0e1831992c6e140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:18.520249 31566 raft_consensus.cc:729] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 6d3a4750a5f94828b0e1831992c6e140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d3a4750a5f94828b0e1831992c6e140, State: Initialized, Role: FOLLOWER
I20260809 07:53:18.520346 31566 consensus_queue.cc:260] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 6d3a4750a5f94828b0e1831992c6e140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } }
I20260809 07:53:18.520561 31566 ts_tablet_manager.cc:1428] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 6d3a4750a5f94828b0e1831992c6e140: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 07:53:18.520628 31574 tablet_replica.cc:439] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 6d3a4750a5f94828b0e1831992c6e140: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } } }
I20260809 07:53:18.520779 31574 tablet_replica.cc:442] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 6d3a4750a5f94828b0e1831992c6e140: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:18.523370 31567 tablet_bootstrap.cc:492] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f: No bootstrap required, opened a new log
I20260809 07:53:18.523473 31567 ts_tablet_manager.cc:1397] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260809 07:53:18.523908 31567 raft_consensus.cc:348] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } }
I20260809 07:53:18.524016 31567 raft_consensus.cc:374] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:18.524046 31567 raft_consensus.cc:729] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53085041937b4ab7bcf7a9cd243fd73f, State: Initialized, Role: FOLLOWER
I20260809 07:53:18.524206 31567 consensus_queue.cc:260] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } }
I20260809 07:53:18.524492 31567 ts_tablet_manager.cc:1428] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:18.524767 31602 tablet_replica.cc:439] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } } }
I20260809 07:53:18.524900 31602 tablet_replica.cc:442] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:18.626632 31602 raft_consensus.cc:482] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:18.626793 31602 raft_consensus.cc:504] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } }
I20260809 07:53:18.627231 31602 leader_election.cc:288] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d3a4750a5f94828b0e1831992c6e140 (127.28.112.1:41629), da1833a5e4b74903a53c461c585ee07c (127.28.112.2:32931)
I20260809 07:53:18.627683 31029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a93f8bcaf64e7eb48b4abf44e43d4b" candidate_uuid: "53085041937b4ab7bcf7a9cd243fd73f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d3a4750a5f94828b0e1831992c6e140" is_pre_election: true
I20260809 07:53:18.627728 31241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a93f8bcaf64e7eb48b4abf44e43d4b" candidate_uuid: "53085041937b4ab7bcf7a9cd243fd73f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da1833a5e4b74903a53c461c585ee07c" is_pre_election: true
I20260809 07:53:18.627859 31241 raft_consensus.cc:2454] T 29a93f8bcaf64e7eb48b4abf44e43d4b P da1833a5e4b74903a53c461c585ee07c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53085041937b4ab7bcf7a9cd243fd73f in term 0.
I20260809 07:53:18.627859 31029 raft_consensus.cc:2454] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 6d3a4750a5f94828b0e1831992c6e140 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53085041937b4ab7bcf7a9cd243fd73f in term 0.
I20260809 07:53:18.628158 31349 leader_election.cc:302] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53085041937b4ab7bcf7a9cd243fd73f, da1833a5e4b74903a53c461c585ee07c; no voters:
I20260809 07:53:18.628332 31602 raft_consensus.cc:2781] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:53:18.628405 31602 raft_consensus.cc:482] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:53:18.628432 31602 raft_consensus.cc:3037] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:18.629420 31602 raft_consensus.cc:504] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } }
I20260809 07:53:18.629736 31602 leader_election.cc:288] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f [CANDIDATE]: Term 1 election: Requested vote from peers 6d3a4750a5f94828b0e1831992c6e140 (127.28.112.1:41629), da1833a5e4b74903a53c461c585ee07c (127.28.112.2:32931)
I20260809 07:53:18.630111 31029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a93f8bcaf64e7eb48b4abf44e43d4b" candidate_uuid: "53085041937b4ab7bcf7a9cd243fd73f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d3a4750a5f94828b0e1831992c6e140"
I20260809 07:53:18.630156 31241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a93f8bcaf64e7eb48b4abf44e43d4b" candidate_uuid: "53085041937b4ab7bcf7a9cd243fd73f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da1833a5e4b74903a53c461c585ee07c"
I20260809 07:53:18.630223 31029 raft_consensus.cc:3037] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 6d3a4750a5f94828b0e1831992c6e140 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:18.630270 31241 raft_consensus.cc:3037] T 29a93f8bcaf64e7eb48b4abf44e43d4b P da1833a5e4b74903a53c461c585ee07c [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:18.631086 31029 raft_consensus.cc:2454] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 6d3a4750a5f94828b0e1831992c6e140 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53085041937b4ab7bcf7a9cd243fd73f in term 1.
I20260809 07:53:18.631177 31241 raft_consensus.cc:2454] T 29a93f8bcaf64e7eb48b4abf44e43d4b P da1833a5e4b74903a53c461c585ee07c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53085041937b4ab7bcf7a9cd243fd73f in term 1.
I20260809 07:53:18.631364 31349 leader_election.cc:302] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53085041937b4ab7bcf7a9cd243fd73f, 6d3a4750a5f94828b0e1831992c6e140; no voters:
I20260809 07:53:18.631523 31602 raft_consensus.cc:2781] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:18.631635 31602 raft_consensus.cc:686] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f [term 1 LEADER]: Becoming Leader. State: Replica: 53085041937b4ab7bcf7a9cd243fd73f, State: Running, Role: LEADER
I20260809 07:53:18.631779 31602 consensus_queue.cc:237] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } }
I20260809 07:53:18.632416 31578 tablet_replica.cc:439] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f: TxnStatusTablet state changed. Reason: New leader 53085041937b4ab7bcf7a9cd243fd73f. Latest consensus state: current_term: 1 leader_uuid: "53085041937b4ab7bcf7a9cd243fd73f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } } }
I20260809 07:53:18.632526 31578 tablet_replica.cc:442] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f: This TxnStatusTablet replica's current role is: LEADER
I20260809 07:53:18.632741 31634 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 07:53:18.632894 31634 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 07:53:18.633432 30798 catalog_manager.cc:5495] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f reported cstate change: term changed from 0 to 1, leader changed from <none> to 53085041937b4ab7bcf7a9cd243fd73f (127.28.112.3). New cstate: current_term: 1 leader_uuid: "53085041937b4ab7bcf7a9cd243fd73f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } health_report { overall_health: HEALTHY } } }
I20260809 07:53:18.644155 31628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.640941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:46055 (local address 127.0.0.1:58836)
0809 07:53:18.641161 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:18.641177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:18.641289 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:18.641528 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:18.641531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:18.641539 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 07:53:18.641649 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.641654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.642411 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.642415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.643017 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.643026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.643455 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.643459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.643551 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.643597 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 07:53:18.643878 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 07:53:18.643893 (+ 15us) client_negotiation.cc:769] Sending connection context
0809 07:53:18.643948 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 07:53:18.644037 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260809 07:53:18.644157 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.641037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58836 (local address 127.28.112.3:46055)
0809 07:53:18.641162 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:18.641167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:18.641220 (+ 53us) server_negotiation.cc:407] Connection header received
0809 07:53:18.641393 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:18.641396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:18.641441 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:18.641501 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:18.641742 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.642298 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.643128 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.643356 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.643445 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.643734 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:18.643803 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:18.643862 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:18.643970 (+ 108us) server_negotiation.cc:299] Negotiation successful
0809 07:53:18.644021 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 07:53:18.646713 31241 raft_consensus.cc:1261] T 29a93f8bcaf64e7eb48b4abf44e43d4b P da1833a5e4b74903a53c461c585ee07c [term 1 FOLLOWER]: Refusing update from remote peer 53085041937b4ab7bcf7a9cd243fd73f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:18.646713 31029 raft_consensus.cc:1261] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 6d3a4750a5f94828b0e1831992c6e140 [term 1 FOLLOWER]: Refusing update from remote peer 53085041937b4ab7bcf7a9cd243fd73f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:18.647147 31578 consensus_queue.cc:1035] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:18.647440 31602 consensus_queue.cc:1035] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f [LEADER]: Connected to new peer: Peer: permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:18.648526 31574 tablet_replica.cc:439] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 6d3a4750a5f94828b0e1831992c6e140: TxnStatusTablet state changed. Reason: New leader 53085041937b4ab7bcf7a9cd243fd73f. Latest consensus state: current_term: 1 leader_uuid: "53085041937b4ab7bcf7a9cd243fd73f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } } }
I20260809 07:53:18.648648 31574 tablet_replica.cc:442] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 6d3a4750a5f94828b0e1831992c6e140: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:18.649001 31576 tablet_replica.cc:439] T 29a93f8bcaf64e7eb48b4abf44e43d4b P da1833a5e4b74903a53c461c585ee07c: TxnStatusTablet state changed. Reason: New leader 53085041937b4ab7bcf7a9cd243fd73f. Latest consensus state: current_term: 1 leader_uuid: "53085041937b4ab7bcf7a9cd243fd73f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } } }
I20260809 07:53:18.649111 31576 tablet_replica.cc:442] T 29a93f8bcaf64e7eb48b4abf44e43d4b P da1833a5e4b74903a53c461c585ee07c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:18.651278 31578 tablet_replica.cc:439] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "53085041937b4ab7bcf7a9cd243fd73f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } } }
I20260809 07:53:18.651376 31578 tablet_replica.cc:442] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f: This TxnStatusTablet replica's current role is: LEADER
I20260809 07:53:18.651898 31576 tablet_replica.cc:439] T 29a93f8bcaf64e7eb48b4abf44e43d4b P da1833a5e4b74903a53c461c585ee07c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "53085041937b4ab7bcf7a9cd243fd73f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } } }
I20260809 07:53:18.651998 31576 tablet_replica.cc:442] T 29a93f8bcaf64e7eb48b4abf44e43d4b P da1833a5e4b74903a53c461c585ee07c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:18.661458 31654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.657918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60754 (local address 127.28.112.2:32931)
0809 07:53:18.658184 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:18.658193 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:18.658206 (+ 13us) server_negotiation.cc:407] Connection header received
0809 07:53:18.658363 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:18.658366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:18.658418 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:18.658520 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:18.658753 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.659256 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.660059 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.660309 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.660395 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.660659 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:18.661074 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:18.661178 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:18.661260 (+ 82us) server_negotiation.cc:299] Negotiation successful
0809 07:53:18.661314 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":127,"threads_started":1}
I20260809 07:53:18.662919 31602 tablet_replica.cc:439] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "53085041937b4ab7bcf7a9cd243fd73f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } } }
I20260809 07:53:18.663045 31602 tablet_replica.cc:442] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 53085041937b4ab7bcf7a9cd243fd73f: This TxnStatusTablet replica's current role is: LEADER
I20260809 07:53:18.663704 31574 tablet_replica.cc:439] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 6d3a4750a5f94828b0e1831992c6e140: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "53085041937b4ab7bcf7a9cd243fd73f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 } } peers { permanent_uuid: "da1833a5e4b74903a53c461c585ee07c" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32931 } } peers { permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 } } }
I20260809 07:53:18.663820 31574 tablet_replica.cc:442] T 29a93f8bcaf64e7eb48b4abf44e43d4b P 6d3a4750a5f94828b0e1831992c6e140: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:18.672260 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.669280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:32791 (local address 127.28.112.3:46055)
0809 07:53:18.669446 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:18.669451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:18.669504 (+ 53us) server_negotiation.cc:407] Connection header received
0809 07:53:18.669638 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:18.669641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:18.669686 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:18.669752 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:18.669996 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.670472 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.671298 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.671510 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.671594 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.671865 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:18.671932 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:18.671987 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:18.672076 (+ 89us) server_negotiation.cc:299] Negotiation successful
0809 07:53:18.672123 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260809 07:53:18.672267 31660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.669210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:46055 (local address 127.28.112.2:32791)
0809 07:53:18.669436 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:18.669450 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:18.669554 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:18.669771 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:18.669774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:18.669782 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 07:53:18.669877 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.669883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.670571 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.670574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.671189 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.671198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.671604 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.671608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.671706 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.671746 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 07:53:18.672004 (+ 258us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 07:53:18.672010 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 07:53:18.672067 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 07:53:18.672145 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":56,"threads_started":1}
I20260809 07:53:18.678710 31029 raft_consensus.cc:1261] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140 [term 1 FOLLOWER]: Refusing update from remote peer da1833a5e4b74903a53c461c585ee07c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:18.678830 31465 raft_consensus.cc:1261] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f [term 1 FOLLOWER]: Refusing update from remote peer da1833a5e4b74903a53c461c585ee07c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:18.679183 31618 consensus_queue.cc:1035] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d3a4750a5f94828b0e1831992c6e140" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:18.679742 31618 consensus_queue.cc:1035] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [LEADER]: Connected to new peer: Peer: permanent_uuid: "53085041937b4ab7bcf7a9cd243fd73f" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 46055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:18.701452 31654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.698077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:38521 (local address 127.28.112.2:32931)
0809 07:53:18.698184 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:18.698188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:18.698200 (+ 12us) server_negotiation.cc:407] Connection header received
0809 07:53:18.698317 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:18.698319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:18.698358 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:18.698409 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:18.698670 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.699333 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.700510 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.700743 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.700820 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.701108 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:18.701171 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:18.701217 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:18.701309 (+ 92us) server_negotiation.cc:299] Negotiation successful
0809 07:53:18.701351 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 07:53:18.701525 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.697813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:32931 (local address 127.28.112.3:38521)
0809 07:53:18.698117 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:18.698133 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:18.698227 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:18.698434 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:18.698436 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:18.698443 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 07:53:18.698573 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.698580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.699434 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.699438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.700402 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:18.700412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.700840 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.700843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:18.700948 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.701000 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 07:53:18.701239 (+ 239us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 07:53:18.701244 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 07:53:18.701303 (+ 59us) client_negotiation.cc:240] Negotiation successful
0809 07:53:18.701384 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":80,"threads_started":1}
I20260809 07:53:18.723901 30797 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:18.724092 30797 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:18.726356 30797 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34b47348a96a4951b8128f7e8e7d6264: Sending DeleteTablet for 3 replicas of tablet 8d0027e2d96443969465a914edb854ca
I20260809 07:53:18.726999 31003 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d0027e2d96443969465a914edb854ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:18 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:52792
I20260809 07:53:18.727238 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d0027e2d96443969465a914edb854ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:18 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:58814
I20260809 07:53:18.727543 31671 tablet_replica.cc:330] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140: stopping tablet replica
I20260809 07:53:18.727528 31212 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d0027e2d96443969465a914edb854ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:18 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:43086
I20260809 07:53:18.727741 31671 raft_consensus.cc:2229] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:18.727969 31671 raft_consensus.cc:2258] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:18.728870 31672 tablet_replica.cc:330] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f: stopping tablet replica
I20260809 07:53:18.728996 31672 raft_consensus.cc:2229] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:18.729110 31672 raft_consensus.cc:2258] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:18.729346 31673 tablet_replica.cc:330] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c: stopping tablet replica
I20260809 07:53:18.729466 31673 raft_consensus.cc:2229] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:53:18.729765 31673 raft_consensus.cc:2258] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:18.729992 31672 ts_tablet_manager.cc:1905] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:18.730409 31673 ts_tablet_manager.cc:1905] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:18.733065 31672 ts_tablet_manager.cc:1918] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 07:53:18.733178 31673 ts_tablet_manager.cc:1918] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 07:53:18.733256 31673 log.cc:1196] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/8d0027e2d96443969465a914edb854ca
I20260809 07:53:18.733532 31673 ts_tablet_manager.cc:1939] T 8d0027e2d96443969465a914edb854ca P da1833a5e4b74903a53c461c585ee07c: Deleting consensus metadata
I20260809 07:53:18.733561 31671 ts_tablet_manager.cc:1905] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:18.734220 30777 catalog_manager.cc:4841] TS da1833a5e4b74903a53c461c585ee07c (127.28.112.2:32931): tablet 8d0027e2d96443969465a914edb854ca (table test-table [id=77f2e56192f04b7881a2735d92d9f772]) successfully deleted
I20260809 07:53:18.736498 31671 ts_tablet_manager.cc:1918] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 07:53:18.736639 31672 log.cc:1196] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/8d0027e2d96443969465a914edb854ca
I20260809 07:53:18.736950 31672 ts_tablet_manager.cc:1939] T 8d0027e2d96443969465a914edb854ca P 53085041937b4ab7bcf7a9cd243fd73f: Deleting consensus metadata
I20260809 07:53:18.737638 30777 catalog_manager.cc:4841] TS 53085041937b4ab7bcf7a9cd243fd73f (127.28.112.3:46055): tablet 8d0027e2d96443969465a914edb854ca (table test-table [id=77f2e56192f04b7881a2735d92d9f772]) successfully deleted
I20260809 07:53:18.737895 31671 log.cc:1196] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786261991120089-29120-0/minicluster-data/ts-0/wal/wals/8d0027e2d96443969465a914edb854ca
I20260809 07:53:18.738240 31671 ts_tablet_manager.cc:1939] T 8d0027e2d96443969465a914edb854ca P 6d3a4750a5f94828b0e1831992c6e140: Deleting consensus metadata
I20260809 07:53:18.738901 30777 catalog_manager.cc:4841] TS 6d3a4750a5f94828b0e1831992c6e140 (127.28.112.1:41629): tablet 8d0027e2d96443969465a914edb854ca (table test-table [id=77f2e56192f04b7881a2735d92d9f772]) successfully deleted
Aug 09 07:53:18 dist-test-slave-vpvm krb5kdc[30730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786261998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.112.1@KRBTEST.COM
I20260809 07:53:18.747049 31565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.734702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43078 (local address 127.28.112.1:41629)
0809 07:53:18.734807 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:18.734812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:18.735371 (+ 559us) server_negotiation.cc:407] Connection header received
0809 07:53:18.735566 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:18.735569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:18.735617 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:18.735666 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:18.736581 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.737330 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.738558 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.738882 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.738951 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.742770 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:18.742779 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:18.742796 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:18.742822 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:18.744735 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:18.745277 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:18.745285 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:18.745287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:18.745386 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:18.745703 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:18.745706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:18.745708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:18.746051 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:18.746291 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:18.746748 (+ 457us) server_negotiation.cc:299] Negotiation successful
0809 07:53:18.746874 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260809 07:53:18.747545 31094 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43078
I20260809 07:53:18.759186 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:18.750347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.28.112.62:43097)
0809 07:53:18.750467 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:18.750471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:18.750643 (+ 172us) server_negotiation.cc:407] Connection header received
0809 07:53:18.750820 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:18.750824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:18.750875 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:18.750926 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:18.751780 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.752565 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.753686 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:18.754047 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:18.754161 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:18.755734 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:18.755738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:18.755741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:18.755768 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:18.757398 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:18.757863 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:18.757867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:18.757868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:18.758043 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:18.758332 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:18.758336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:18.758337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:18.758566 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:18.758702 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:18.758922 (+ 220us) server_negotiation.cc:299] Negotiation successful
0809 07:53:18.759037 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260809 07:53:18.759523 30797 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49620
I20260809 07:53:18.760984 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 30900
I20260809 07:53:18.783444 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 31120
I20260809 07:53:18.793581 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 31323
I20260809 07:53:18.804559 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 30748
2026-08-09T07:53:18Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2924 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:53:18 dist-test-slave-vpvm krb5kdc[31680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:53:18 dist-test-slave-vpvm krb5kdc[31680](info): set up 2 sockets
Aug 09 07:53:18 dist-test-slave-vpvm krb5kdc[31680](info): commencing operation
krb5kdc: starting...
W20260809 07:53:19.960464 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:53:20 dist-test-slave-vpvm krb5kdc[31680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:53:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:53:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:53:20.070083 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:46557
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:46557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:20.175758 31700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:20.175997 31700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:20.176059 31700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:20.179965 31700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:20.180018 31700 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 07:53:20.180048 31700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:20.180089 31700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:20.180119 31700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:20.180147 31700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:53:20.185206 31700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:46557
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:46557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:20.186204 31700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:20.187145 31700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 07:53:20.195163 31700 server_base.cc:1034] running on GCE node
W20260809 07:53:20.195122 31711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:20.195113 31709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:20.195114 31708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:20.195863 31700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:20.196836 31700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:20.198000 31700 hybrid_clock.cc:648] HybridClock initialized: now 1786262000197984 us; error 25 us; skew 500 ppm
Aug 09 07:53:20 dist-test-slave-vpvm krb5kdc[31680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262000, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:20.200448 31700 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:53:20.201270 31700 webserver.cc:460] Webserver started at http://127.28.112.62:37953/ using document root <none> and password file <none>
I20260809 07:53:20.201769 31700 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:20.201834 31700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:20.202060 31700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:20.203963 31700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "a01ba57d6e004e4991c7a10dd11dce14"
format_stamp: "Formatted at 2026-08-09 07:53:20 on dist-test-slave-vpvm"
server_key: "141b15039b938ce85e290f5fde690cbd"
server_key_iv: "74e735f0cbca5fdae11d0c9970ab392b"
server_key_version: "encryptionkey@0"
I20260809 07:53:20.204495 31700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "a01ba57d6e004e4991c7a10dd11dce14"
format_stamp: "Formatted at 2026-08-09 07:53:20 on dist-test-slave-vpvm"
server_key: "141b15039b938ce85e290f5fde690cbd"
server_key_iv: "74e735f0cbca5fdae11d0c9970ab392b"
server_key_version: "encryptionkey@0"
I20260809 07:53:20.207918 31700 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260809 07:53:20.210070 31721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:20.211174 31700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 07:53:20.211287 31700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "a01ba57d6e004e4991c7a10dd11dce14"
format_stamp: "Formatted at 2026-08-09 07:53:20 on dist-test-slave-vpvm"
server_key: "141b15039b938ce85e290f5fde690cbd"
server_key_iv: "74e735f0cbca5fdae11d0c9970ab392b"
server_key_version: "encryptionkey@0"
I20260809 07:53:20.211375 31700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:20.227578 31700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:20.229909 31700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:20.230084 31700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:20.237356 31700 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:46557
I20260809 07:53:20.237378 31826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:46557 every 8 connection(s)
I20260809 07:53:20.238232 31700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:53:20.240875 31827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:20.244968 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 31700
I20260809 07:53:20.245066 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:53:20.245299 29120 external_mini_cluster.cc:1428] Setting key 3e313f29b1b9a6c274032575f4432697
I20260809 07:53:20.246316 31827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14: Bootstrap starting.
I20260809 07:53:20.249014 31827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:20.249915 31827 log.cc:824] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:20.252116 31827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14: No bootstrap required, opened a new log
Aug 09 07:53:20 dist-test-slave-vpvm krb5kdc[31680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:20.256315 31827 raft_consensus.cc:348] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a01ba57d6e004e4991c7a10dd11dce14" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 46557 } }
I20260809 07:53:20.256533 31827 raft_consensus.cc:374] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:20.256637 31827 raft_consensus.cc:729] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a01ba57d6e004e4991c7a10dd11dce14, State: Initialized, Role: FOLLOWER
I20260809 07:53:20.257088 31827 consensus_queue.cc:260] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a01ba57d6e004e4991c7a10dd11dce14" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 46557 } }
I20260809 07:53:20.257222 31827 raft_consensus.cc:388] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:53:20.257285 31827 raft_consensus.cc:482] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:53:20.257383 31827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:20.258360 31827 raft_consensus.cc:504] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a01ba57d6e004e4991c7a10dd11dce14" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 46557 } }
I20260809 07:53:20.258709 31827 leader_election.cc:302] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a01ba57d6e004e4991c7a10dd11dce14; no voters:
I20260809 07:53:20.259006 31827 leader_election.cc:288] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:53:20.259106 31834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:20.259311 31834 raft_consensus.cc:686] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14 [term 1 LEADER]: Becoming Leader. State: Replica: a01ba57d6e004e4991c7a10dd11dce14, State: Running, Role: LEADER
I20260809 07:53:20.259598 31834 consensus_queue.cc:237] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a01ba57d6e004e4991c7a10dd11dce14" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 46557 } }
I20260809 07:53:20.260283 31827 sys_catalog.cc:564] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:20.261144 31835 sys_catalog.cc:455] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a01ba57d6e004e4991c7a10dd11dce14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a01ba57d6e004e4991c7a10dd11dce14" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 46557 } } }
I20260809 07:53:20.261267 31835 sys_catalog.cc:458] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:20.261305 31836 sys_catalog.cc:455] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a01ba57d6e004e4991c7a10dd11dce14. Latest consensus state: current_term: 1 leader_uuid: "a01ba57d6e004e4991c7a10dd11dce14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a01ba57d6e004e4991c7a10dd11dce14" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 46557 } } }
I20260809 07:53:20.261376 31836 sys_catalog.cc:458] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:20.261642 31839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:53:20.262802 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:20.246586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.28.112.62:46557)
0809 07:53:20.246983 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:20.247001 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:20.247041 (+ 40us) server_negotiation.cc:407] Connection header received
0809 07:53:20.247855 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:20.247878 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:20.248267 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:20.248628 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:20.249742 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.251219 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.252414 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.253218 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.253374 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:20.257189 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:20.257195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:20.257210 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:20.257241 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:20.259059 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:20.259596 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:20.259602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:20.259607 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:20.259740 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:20.260062 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:20.260065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:20.260067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:20.260546 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:20.261384 (+ 838us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:20.261574 (+ 190us) server_negotiation.cc:299] Negotiation successful
0809 07:53:20.261867 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":96,"threads_started":1}
I20260809 07:53:20.264659 31839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:20.270036 31839 catalog_manager.cc:1347] Generated new cluster ID: 7fc4a8e8bb934eac9a40bb977d11884e
I20260809 07:53:20.270090 31839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:20.279333 31839 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:53:20.280151 31839 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:20.290308 31839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14: Generated new TSK 0
I20260809 07:53:20.290915 31839 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
I20260809 07:53:20.322971 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:46557
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:20.461673 31869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:20.461896 31869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:20.461979 31869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:20.465942 31869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:20.466014 31869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:20.466068 31869 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 07:53:20.466094 31869 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 07:53:20.466138 31869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:53:20.471163 31869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.112.62:46557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:20.472092 31869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:20.473034 31869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:20.481819 31876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:20.481911 31869 server_base.cc:1034] running on GCE node
W20260809 07:53:20.482193 31877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:20.481840 31879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:20.482637 31869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:20.483245 31869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:20.484398 31869 hybrid_clock.cc:648] HybridClock initialized: now 1786262000484382 us; error 29 us; skew 500 ppm
Aug 09 07:53:20 dist-test-slave-vpvm krb5kdc[31680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262000, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:20.486968 31869 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:53:20.487762 31869 webserver.cc:460] Webserver started at http://127.28.112.1:41341/ using document root <none> and password file <none>
I20260809 07:53:20.488274 31869 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:20.488338 31869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:20.488533 31869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:20.490489 31869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "7ce9b38adfe24c2882b8cfbec30903cb"
format_stamp: "Formatted at 2026-08-09 07:53:20 on dist-test-slave-vpvm"
server_key: "2f3a901a1338766b83924ff4a96c4aef"
server_key_iv: "37e34eb00e87b620ada6700f5aca7f56"
server_key_version: "encryptionkey@0"
I20260809 07:53:20.490996 31869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "7ce9b38adfe24c2882b8cfbec30903cb"
format_stamp: "Formatted at 2026-08-09 07:53:20 on dist-test-slave-vpvm"
server_key: "2f3a901a1338766b83924ff4a96c4aef"
server_key_iv: "37e34eb00e87b620ada6700f5aca7f56"
server_key_version: "encryptionkey@0"
I20260809 07:53:20.494629 31869 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 07:53:20.496769 31890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:20.497870 31869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:20.498004 31869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "7ce9b38adfe24c2882b8cfbec30903cb"
format_stamp: "Formatted at 2026-08-09 07:53:20 on dist-test-slave-vpvm"
server_key: "2f3a901a1338766b83924ff4a96c4aef"
server_key_iv: "37e34eb00e87b620ada6700f5aca7f56"
server_key_version: "encryptionkey@0"
I20260809 07:53:20.498101 31869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:20.520556 31869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:20.523478 31869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:20.523649 31869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:20.524986 31869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:20.525044 31869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:20.525100 31869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:20.525125 31869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 07:53:20 dist-test-slave-vpvm krb5kdc[31680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262000, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:20.535295 31869 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:36585
I20260809 07:53:20.535403 32070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:36585 every 8 connection(s)
I20260809 07:53:20.536331 31869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:53:20.538870 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:20.525743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:56819 (local address 127.28.112.62:46557)
0809 07:53:20.525889 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:20.525893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:20.526821 (+ 928us) server_negotiation.cc:407] Connection header received
0809 07:53:20.527970 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:20.527974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:20.528035 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:20.528130 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:20.529605 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.530318 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.531945 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.532246 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.532342 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:20.534856 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:20.534860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:20.534863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:20.534890 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:20.536522 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:20.537220 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:20.537223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:20.537225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:20.537323 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:20.537706 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:20.537709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:20.537711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:20.537928 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:20.538041 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:20.538640 (+ 599us) server_negotiation.cc:299] Negotiation successful
0809 07:53:20.538752 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 07:53:20.539463 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 31869
I20260809 07:53:20.539585 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:53:20.539758 31910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:20.526160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:46557 (local address 127.28.112.1:56819)
0809 07:53:20.526648 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:20.526705 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:20.527727 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:20.528330 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:20.528340 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:20.528795 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:20.529438 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:20.529458 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.530433 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.530437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:20.531815 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:20.531826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.532409 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.532413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:20.532600 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:20.533091 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:20.533108 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:20.534700 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:20.536668 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:20.536677 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:20.536691 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:20.537078 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:20.537428 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:20.537432 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:20.537434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:20.537596 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:20.538039 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:20.538046 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:20.538373 (+ 327us) client_negotiation.cc:769] Sending connection context
0809 07:53:20.538581 (+ 208us) client_negotiation.cc:240] Negotiation successful
0809 07:53:20.538828 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":109,"threads_started":1}
I20260809 07:53:20.539894 29120 external_mini_cluster.cc:1428] Setting key 0510ba3039125c41a9b865de834660c5
I20260809 07:53:20.540966 32072 heartbeater.cc:347] Connected to a master server at 127.28.112.62:46557
I20260809 07:53:20.541205 32072 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:20.541692 32072 heartbeater.cc:511] Master 127.28.112.62:46557 requested a full tablet report, sending...
I20260809 07:53:20.544204 31750 ts_manager.cc:194] Registered new tserver with Master: 7ce9b38adfe24c2882b8cfbec30903cb (127.28.112.1:36585)
I20260809 07:53:20.545711 31750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:56819
I20260809 07:53:20.549741 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:20.543374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:32861 (local address 127.28.112.62:46557)
0809 07:53:20.543489 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:20.543494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:20.543696 (+ 202us) server_negotiation.cc:407] Connection header received
0809 07:53:20.543784 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:20.543787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:20.543829 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:20.543891 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:20.544286 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.544819 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.545603 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.545831 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.545918 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:20.547063 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:20.547067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:20.547069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:20.547094 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:20.548319 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:20.548691 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:20.548694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:20.548696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:20.548768 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:20.549005 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:20.549008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:20.549010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:20.549172 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:20.549255 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:20.549493 (+ 238us) server_negotiation.cc:299] Negotiation successful
0809 07:53:20.549602 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 07:53:20.549750 32082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:20.543302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:46557 (local address 127.28.112.1:32861)
0809 07:53:20.543618 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:20.543630 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:20.543716 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:20.543915 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:20.543918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:20.544095 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:20.544185 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:20.544190 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.544944 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.544948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:20.545497 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:20.545505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.546047 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.546051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:20.546121 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:20.546555 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:20.546565 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:20.546943 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:20.548418 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:20.548421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:20.548423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:20.548606 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:20.548847 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:20.548850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:20.548852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:20.548929 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:20.549259 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:20.549262 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:20.549348 (+ 86us) client_negotiation.cc:769] Sending connection context
0809 07:53:20.549431 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 07:53:20.549562 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.28.112.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
I20260809 07:53:20.568501 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:46557
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:20.677884 32086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:20.678155 32086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:20.678200 32086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:20.682384 32086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:20.682443 32086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:20.682500 32086 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 07:53:20.682528 32086 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 07:53:20.682577 32086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:53:20.687717 32086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.112.62:46557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:20.688700 32086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:20.689659 32086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:20.698668 32100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:20.698766 32102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:20.698691 32096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:20.698877 32086 server_base.cc:1034] running on GCE node
I20260809 07:53:20.699319 32086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:20.699905 32086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:20.701061 32086 hybrid_clock.cc:648] HybridClock initialized: now 1786262000701033 us; error 37 us; skew 500 ppm
Aug 09 07:53:20 dist-test-slave-vpvm krb5kdc[31680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262000, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:20.703719 32086 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:53:20.704493 32086 webserver.cc:460] Webserver started at http://127.28.112.2:40835/ using document root <none> and password file <none>
I20260809 07:53:20.705003 32086 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:20.705075 32086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:20.705271 32086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:20.707171 32086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042"
format_stamp: "Formatted at 2026-08-09 07:53:20 on dist-test-slave-vpvm"
server_key: "5050e944b54a777c51d2a12ff6cefeb9"
server_key_iv: "2d5f4965f34430ca074bf56df15fb2a8"
server_key_version: "encryptionkey@0"
I20260809 07:53:20.707681 32086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042"
format_stamp: "Formatted at 2026-08-09 07:53:20 on dist-test-slave-vpvm"
server_key: "5050e944b54a777c51d2a12ff6cefeb9"
server_key_iv: "2d5f4965f34430ca074bf56df15fb2a8"
server_key_version: "encryptionkey@0"
I20260809 07:53:20.711046 32086 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:53:20.713155 32114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:20.714293 32086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:20.714394 32086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042"
format_stamp: "Formatted at 2026-08-09 07:53:20 on dist-test-slave-vpvm"
server_key: "5050e944b54a777c51d2a12ff6cefeb9"
server_key_iv: "2d5f4965f34430ca074bf56df15fb2a8"
server_key_version: "encryptionkey@0"
I20260809 07:53:20.714498 32086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:20.730049 32086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:20.732815 32086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:20.732971 32086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:20.734345 32086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:20.734392 32086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:20.734441 32086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:20.734472 32086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 07:53:20 dist-test-slave-vpvm krb5kdc[31680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262000, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:20.744589 32086 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:34431
I20260809 07:53:20.744756 32298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:34431 every 8 connection(s)
I20260809 07:53:20.745648 32086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:53:20.747577 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:20.735038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:33007 (local address 127.28.112.62:46557)
0809 07:53:20.735156 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:20.735160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:20.736121 (+ 961us) server_negotiation.cc:407] Connection header received
0809 07:53:20.737067 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:20.737071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:20.737131 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:20.737217 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:20.738734 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.739324 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.740585 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.740813 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.740901 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:20.744117 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:20.744122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:20.744125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:20.744177 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:20.745655 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:20.746250 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:20.746254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:20.746256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:20.746349 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:20.746705 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:20.746707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:20.746708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:20.746878 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:20.746951 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:20.747351 (+ 400us) server_negotiation.cc:299] Negotiation successful
0809 07:53:20.747445 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 07:53:20.748315 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:20.735525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:46557 (local address 127.28.112.2:33007)
0809 07:53:20.735978 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:20.736018 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:20.736866 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:20.737425 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:20.737435 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:20.737884 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:20.738549 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:20.738572 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.739452 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.739455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:20.740460 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:20.740471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.740967 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.740971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:20.741128 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:20.741681 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:20.741703 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:20.743942 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:20.745785 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:20.745791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:20.745806 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:20.746127 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:20.746447 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:20.746450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:20.746452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:20.746615 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:20.746953 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:20.746958 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:20.747168 (+ 210us) client_negotiation.cc:769] Sending connection context
0809 07:53:20.747307 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 07:53:20.747534 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":138,"threads_started":1}
I20260809 07:53:20.749476 32299 heartbeater.cc:347] Connected to a master server at 127.28.112.62:46557
I20260809 07:53:20.749616 32299 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:20.750108 32299 heartbeater.cc:511] Master 127.28.112.62:46557 requested a full tablet report, sending...
I20260809 07:53:20.751075 31752 ts_manager.cc:194] Registered new tserver with Master: 0f5f4ab9c0fe4d65b67c36f0cd009042 (127.28.112.2:34431)
I20260809 07:53:20.751776 31752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:33007
I20260809 07:53:20.754527 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 32086
I20260809 07:53:20.754630 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:53:20.754895 29120 external_mini_cluster.cc:1428] Setting key 7a7ac36e9f605d567bf88b05dce4d493
I20260809 07:53:20.757900 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:20.751547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:41365 (local address 127.28.112.62:46557)
0809 07:53:20.751648 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:20.751652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:20.751873 (+ 221us) server_negotiation.cc:407] Connection header received
0809 07:53:20.751947 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:20.751950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:20.751994 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:20.752060 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:20.752500 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.753222 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.753980 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.754203 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.754310 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:20.755215 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:20.755219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:20.755222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:20.755247 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:20.756460 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:20.756827 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:20.756830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:20.756832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:20.756901 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:20.757149 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:20.757152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:20.757154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:20.757342 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:20.757426 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:20.757668 (+ 242us) server_negotiation.cc:299] Negotiation successful
0809 07:53:20.757762 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 07:53:20.757915 32312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:20.751494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:46557 (local address 127.28.112.2:41365)
0809 07:53:20.751769 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:20.751780 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:20.751868 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:20.752088 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:20.752091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:20.752290 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:20.752396 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:20.752402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.753326 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.753329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:20.753865 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:20.753872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.754298 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.754300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:20.754364 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:20.754757 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:20.754768 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:20.755088 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:20.756562 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:20.756565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:20.756568 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:20.756742 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:20.756981 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:20.756985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:20.756987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:20.757068 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:20.757435 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:20.757438 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:20.757518 (+ 80us) client_negotiation.cc:769] Sending connection context
0809 07:53:20.757606 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 07:53:20.757735 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.28.112.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
I20260809 07:53:20.779983 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:46557
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:20.889389 32316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:20.889619 32316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:20.889669 32316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:20.893517 32316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:20.893565 32316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:20.893601 32316 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 07:53:20.893620 32316 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 07:53:20.893651 32316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:53:20.898643 32316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.112.62:46557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:20.899530 32316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:20.900440 32316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:20.909672 32329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:20.909672 32327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:20.909653 32324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:20.909720 32316 server_base.cc:1034] running on GCE node
I20260809 07:53:20.910266 32316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:20.910840 32316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:20.911983 32316 hybrid_clock.cc:648] HybridClock initialized: now 1786262000911959 us; error 40 us; skew 500 ppm
Aug 09 07:53:20 dist-test-slave-vpvm krb5kdc[31680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262000, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:20.914507 32316 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:53:20.915351 32316 webserver.cc:460] Webserver started at http://127.28.112.3:35445/ using document root <none> and password file <none>
I20260809 07:53:20.915868 32316 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:20.915933 32316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:20.916113 32316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:20.918058 32316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "014dc17a9cbe4665a8571a3626c781f2"
format_stamp: "Formatted at 2026-08-09 07:53:20 on dist-test-slave-vpvm"
server_key: "ace432d08b4ad348f8fe1bf357295943"
server_key_iv: "278b23f331138357bde79613b9056176"
server_key_version: "encryptionkey@0"
I20260809 07:53:20.918578 32316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "014dc17a9cbe4665a8571a3626c781f2"
format_stamp: "Formatted at 2026-08-09 07:53:20 on dist-test-slave-vpvm"
server_key: "ace432d08b4ad348f8fe1bf357295943"
server_key_iv: "278b23f331138357bde79613b9056176"
server_key_version: "encryptionkey@0"
I20260809 07:53:20.921923 32316 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 07:53:20.924160 32338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:20.925243 32316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:20.925357 32316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "014dc17a9cbe4665a8571a3626c781f2"
format_stamp: "Formatted at 2026-08-09 07:53:20 on dist-test-slave-vpvm"
server_key: "ace432d08b4ad348f8fe1bf357295943"
server_key_iv: "278b23f331138357bde79613b9056176"
server_key_version: "encryptionkey@0"
I20260809 07:53:20.925446 32316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:20.946899 32316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:20.949807 32316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:20.949990 32316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:20.951371 32316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:20.951419 32316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:20.951470 32316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:20.951498 32316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 07:53:20 dist-test-slave-vpvm krb5kdc[31680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262000, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:20.961369 32316 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:36449
I20260809 07:53:20.961438 32513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:36449 every 8 connection(s)
I20260809 07:53:20.962486 32316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:53:20.965075 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:20.952120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:37711 (local address 127.28.112.62:46557)
0809 07:53:20.952263 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:20.952268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:20.953090 (+ 822us) server_negotiation.cc:407] Connection header received
0809 07:53:20.954251 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:20.954255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:20.954313 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:20.954401 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:20.955892 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.956704 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.958358 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.958622 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.958710 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:20.961492 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:20.961496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:20.961498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:20.961525 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:20.963007 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:20.963654 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:20.963658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:20.963660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:20.963740 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:20.964049 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:20.964052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:20.964054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:20.964232 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:20.964318 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:20.964785 (+ 467us) server_negotiation.cc:299] Negotiation successful
0809 07:53:20.964902 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 07:53:20.965794 32349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:20.952477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:46557 (local address 127.28.112.3:37711)
0809 07:53:20.952933 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:20.952981 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:20.953983 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:20.954614 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:20.954624 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:20.955059 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:20.955704 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:20.955725 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.956835 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.956839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:20.958237 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:20.958250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.958782 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.958786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:20.958980 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:20.959551 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:20.959572 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:20.961315 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:20.963120 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:20.963126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:20.963140 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:20.963560 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:20.963821 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:20.963824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:20.963826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:20.963952 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:20.964330 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:20.964336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:20.964565 (+ 229us) client_negotiation.cc:769] Sending connection context
0809 07:53:20.964724 (+ 159us) client_negotiation.cc:240] Negotiation successful
0809 07:53:20.965014 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":130,"threads_started":1}
I20260809 07:53:20.965847 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 32316
I20260809 07:53:20.965975 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:53:20.966248 29120 external_mini_cluster.cc:1428] Setting key 86ce18faa160f962d2d431d97d037369
I20260809 07:53:20.967214 32515 heartbeater.cc:347] Connected to a master server at 127.28.112.62:46557
I20260809 07:53:20.967368 32515 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:20.967927 32515 heartbeater.cc:511] Master 127.28.112.62:46557 requested a full tablet report, sending...
I20260809 07:53:20.968909 31750 ts_manager.cc:194] Registered new tserver with Master: 014dc17a9cbe4665a8571a3626c781f2 (127.28.112.3:36449)
I20260809 07:53:20.969506 31750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:37711
I20260809 07:53:20.975399 32521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:20.969156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:46557 (local address 127.28.112.3:36695)
0809 07:53:20.969865 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:20.969878 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:20.969983 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:20.970212 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:20.970215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:20.970417 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:20.970522 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:20.970529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.971210 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.971214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:20.971800 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:20.971809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.972150 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.972152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:20.972214 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:20.972585 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:20.972595 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:20.972877 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:20.974349 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:20.974351 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:20.974352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:20.974514 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:20.974712 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:20.974714 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:20.974715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:20.974775 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:20.975040 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:20.975043 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:20.975108 (+ 65us) client_negotiation.cc:769] Sending connection context
0809 07:53:20.975178 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 07:53:20.975271 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":89,"threads_started":1}
I20260809 07:53:20.975409 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:20.969231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:36695 (local address 127.28.112.62:46557)
0809 07:53:20.969348 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:20.969352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:20.969939 (+ 587us) server_negotiation.cc:407] Connection header received
0809 07:53:20.970066 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:20.970068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:20.970109 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:20.970181 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:20.970624 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.971090 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.971889 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.972081 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.972148 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:20.972992 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:20.972995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:20.972997 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:20.973018 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:20.974271 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:20.974587 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:20.974589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:20.974590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:20.974655 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:20.974831 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:20.974833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:20.974834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:20.974976 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:20.975048 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:20.975221 (+ 173us) server_negotiation.cc:299] Negotiation successful
0809 07:53:20.975304 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 07:53:20.979655 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 07:53:20 dist-test-slave-vpvm krb5kdc[31680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 07:53:20 dist-test-slave-vpvm krb5kdc[31680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:21.004076 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:20.996810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48770 (local address 127.28.112.62:46557)
0809 07:53:20.996917 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:20.996921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:20.996933 (+ 12us) server_negotiation.cc:407] Connection header received
0809 07:53:20.996972 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:20.996974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:20.997013 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:20.997086 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:20.997722 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.998286 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.999044 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:20.999253 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:20.999327 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.001505 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:21.001509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:21.001511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:21.001542 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:21.002819 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:21.003188 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:21.003190 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:21.003192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:21.003259 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:21.003435 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:21.003437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:21.003438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:21.003606 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:21.003699 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:21.003858 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 07:53:21.003952 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 07:53:21.006729 31750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:53:21.009573 31750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:53:21.023334 32543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.018303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53368 (local address 127.28.112.1:36585)
0809 07:53:21.018900 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:21.018915 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:21.018936 (+ 21us) server_negotiation.cc:407] Connection header received
0809 07:53:21.019009 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:21.019014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:21.019205 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:21.019360 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:21.019648 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.020766 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.022133 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.022682 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.022786 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.022958 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:21.023080 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 07:53:21.023185 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":72,"threads_started":1}
I20260809 07:53:21.023325 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.017591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:36585 (local address 127.0.0.1:53368)
0809 07:53:21.018462 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:21.018475 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:21.018574 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:21.019362 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:21.019368 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:21.019390 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:21.019534 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.019542 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.020880 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.020883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.022010 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.022018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.022802 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.022805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.022909 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.022947 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 07:53:21.023051 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 07:53:21.023181 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":4,"spinlock_wait_cycles":6016,"thread_start_us":119,"threads_started":1}
I20260809 07:53:21.023538 32536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.017457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:34431 (local address 127.0.0.1:57090)
0809 07:53:21.018335 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:21.018386 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:21.018560 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:21.019392 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:21.019394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:21.019412 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:21.019547 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.019554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.020572 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.020576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.021828 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.021839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.023120 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.023124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.023228 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.023254 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 07:53:21.023323 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 07:53:21.023389 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
I20260809 07:53:21.023834 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.018185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35100 (local address 127.28.112.3:36449)
0809 07:53:21.018733 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:21.018742 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:21.019134 (+ 392us) server_negotiation.cc:407] Connection header received
0809 07:53:21.019212 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:21.019217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:21.019386 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:21.019536 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:21.019881 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.020940 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.022189 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.022926 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.023030 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.023177 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:21.023608 (+ 431us) server_negotiation.cc:299] Negotiation successful
0809 07:53:21.023702 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":118,"threads_started":1}
I20260809 07:53:21.023541 32541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.017882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:36449 (local address 127.0.0.1:35100)
0809 07:53:21.018945 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:21.018958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:21.019046 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:21.019677 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:21.019680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:21.019696 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:21.019766 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.019771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.021041 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.021044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.022070 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.022082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.023231 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.023234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.023319 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.023338 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 07:53:21.023392 (+ 54us) client_negotiation.cc:240] Negotiation successful
0809 07:53:21.023440 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"mutex_wait_us":21,"thread_start_us":126,"threads_started":1}
I20260809 07:53:21.024011 32538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.017598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57090 (local address 127.28.112.2:34431)
0809 07:53:21.018324 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:21.018336 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:21.018704 (+ 368us) server_negotiation.cc:407] Connection header received
0809 07:53:21.018790 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:21.018796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:21.019101 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:21.019245 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:21.019674 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.020457 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.021988 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.022798 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.022904 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.023056 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:21.023794 (+ 738us) server_negotiation.cc:299] Negotiation successful
0809 07:53:21.023882 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":99,"threads_started":1}
I20260809 07:53:21.026532 32202 tablet_service.cc:1467] Processing CreateTablet for tablet ebd40c7657e34e3b9bd7592f708af18d (DEFAULT_TABLE table=test-table [id=8939059299ec4764bbfc5ad17fc3ecdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:21.026600 31965 tablet_service.cc:1467] Processing CreateTablet for tablet ebd40c7657e34e3b9bd7592f708af18d (DEFAULT_TABLE table=test-table [id=8939059299ec4764bbfc5ad17fc3ecdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:21.027086 32409 tablet_service.cc:1467] Processing CreateTablet for tablet ebd40c7657e34e3b9bd7592f708af18d (DEFAULT_TABLE table=test-table [id=8939059299ec4764bbfc5ad17fc3ecdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:21.027487 32202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd40c7657e34e3b9bd7592f708af18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:21.027611 31965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd40c7657e34e3b9bd7592f708af18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:21.027977 32409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd40c7657e34e3b9bd7592f708af18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:21.034240 32544 tablet_bootstrap.cc:492] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042: Bootstrap starting.
I20260809 07:53:21.035898 32547 tablet_bootstrap.cc:492] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2: Bootstrap starting.
I20260809 07:53:21.036366 32544 tablet_bootstrap.cc:654] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:21.037062 32544 log.cc:824] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:21.037583 32547 tablet_bootstrap.cc:654] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:21.037639 32550 tablet_bootstrap.cc:492] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb: Bootstrap starting.
I20260809 07:53:21.038399 32547 log.cc:824] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:21.039122 32544 tablet_bootstrap.cc:492] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042: No bootstrap required, opened a new log
I20260809 07:53:21.039371 32544 ts_tablet_manager.cc:1397] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 07:53:21.039685 32550 tablet_bootstrap.cc:654] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:21.040315 32547 tablet_bootstrap.cc:492] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2: No bootstrap required, opened a new log
I20260809 07:53:21.040393 32550 log.cc:824] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:21.040498 32547 ts_tablet_manager.cc:1397] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 07:53:21.041936 32550 tablet_bootstrap.cc:492] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb: No bootstrap required, opened a new log
I20260809 07:53:21.042243 32550 ts_tablet_manager.cc:1397] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260809 07:53:21.043246 32544 raft_consensus.cc:348] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.043618 32544 raft_consensus.cc:374] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:21.043715 32544 raft_consensus.cc:729] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5f4ab9c0fe4d65b67c36f0cd009042, State: Initialized, Role: FOLLOWER
I20260809 07:53:21.044267 32544 consensus_queue.cc:260] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.044194 32547 raft_consensus.cc:348] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.044418 32547 raft_consensus.cc:374] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:21.044495 32547 raft_consensus.cc:729] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014dc17a9cbe4665a8571a3626c781f2, State: Initialized, Role: FOLLOWER
I20260809 07:53:21.044979 32547 consensus_queue.cc:260] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.045735 32544 ts_tablet_manager.cc:1428] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 07:53:21.045814 32547 ts_tablet_manager.cc:1428] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 07:53:21.046128 32515 heartbeater.cc:503] Master 127.28.112.62:46557 was elected leader, sending a full tablet report...
I20260809 07:53:21.046059 32550 raft_consensus.cc:348] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.046303 32550 raft_consensus.cc:374] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:21.046375 32550 raft_consensus.cc:729] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce9b38adfe24c2882b8cfbec30903cb, State: Initialized, Role: FOLLOWER
I20260809 07:53:21.046743 32550 consensus_queue.cc:260] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.047086 32299 heartbeater.cc:503] Master 127.28.112.62:46557 was elected leader, sending a full tablet report...
I20260809 07:53:21.047336 32072 heartbeater.cc:503] Master 127.28.112.62:46557 was elected leader, sending a full tablet report...
I20260809 07:53:21.047706 32550 ts_tablet_manager.cc:1428] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
W20260809 07:53:21.213475 32516 tablet.cc:2369] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:53:21.237397 32562 raft_consensus.cc:482] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:21.237636 32562 raft_consensus.cc:504] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.238986 32562 leader_election.cc:288] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f5f4ab9c0fe4d65b67c36f0cd009042 (127.28.112.2:34431), 014dc17a9cbe4665a8571a3626c781f2 (127.28.112.3:36449)
I20260809 07:53:21.243500 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.239542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:34207 (local address 127.28.112.3:36449)
0809 07:53:21.239651 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:21.239655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:21.239926 (+ 271us) server_negotiation.cc:407] Connection header received
0809 07:53:21.240114 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:21.240117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:21.240174 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:21.240217 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:21.240605 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.241261 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.242349 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.242887 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.243003 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.243057 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:21.243293 (+ 236us) server_negotiation.cc:299] Negotiation successful
0809 07:53:21.243374 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 07:53:21.243662 32538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.239431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:56561 (local address 127.28.112.2:34431)
0809 07:53:21.239572 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:21.239577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:21.239801 (+ 224us) server_negotiation.cc:407] Connection header received
0809 07:53:21.239939 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:21.239942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:21.239983 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:21.240053 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:21.240303 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.240976 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.242561 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.243137 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.243254 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.243302 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:21.243483 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 07:53:21.243552 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 07:53:21.243680 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.239214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:34431 (local address 127.28.112.1:56561)
0809 07:53:21.239710 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:21.239721 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:21.239834 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:21.240065 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:21.240067 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:21.240085 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:21.240203 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.240209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.241084 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.241087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.242441 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.242452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.243254 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.243258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.243369 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.243391 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 07:53:21.243458 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 07:53:21.243525 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":29,"thread_start_us":86,"threads_started":1}
I20260809 07:53:21.244000 32564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.239214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:36449 (local address 127.28.112.1:34207)
0809 07:53:21.239832 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:21.239847 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:21.240004 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:21.240346 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:21.240349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:21.240364 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:21.240481 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.240486 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.241365 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.241369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.242235 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.242245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.243020 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.243023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.243120 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.243168 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 07:53:21.243786 (+ 618us) client_negotiation.cc:240] Negotiation successful
0809 07:53:21.243861 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":187,"thread_start_us":77,"threads_started":1}
I20260809 07:53:21.244369 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd40c7657e34e3b9bd7592f708af18d" candidate_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" is_pre_election: true
I20260809 07:53:21.244576 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd40c7657e34e3b9bd7592f708af18d" candidate_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014dc17a9cbe4665a8571a3626c781f2" is_pre_election: true
I20260809 07:53:21.244683 32228 raft_consensus.cc:2454] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ce9b38adfe24c2882b8cfbec30903cb in term 0.
I20260809 07:53:21.244843 32436 raft_consensus.cc:2454] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ce9b38adfe24c2882b8cfbec30903cb in term 0.
I20260809 07:53:21.245187 31897 leader_election.cc:302] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f5f4ab9c0fe4d65b67c36f0cd009042, 7ce9b38adfe24c2882b8cfbec30903cb; no voters:
I20260809 07:53:21.245440 32562 raft_consensus.cc:2781] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:53:21.245515 32562 raft_consensus.cc:482] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:53:21.245566 32562 raft_consensus.cc:3037] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:21.246766 32562 raft_consensus.cc:504] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
W20260809 07:53:21.246865 32300 tablet.cc:2369] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:53:21.247123 32562 leader_election.cc:288] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [CANDIDATE]: Term 1 election: Requested vote from peers 0f5f4ab9c0fe4d65b67c36f0cd009042 (127.28.112.2:34431), 014dc17a9cbe4665a8571a3626c781f2 (127.28.112.3:36449)
I20260809 07:53:21.247546 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd40c7657e34e3b9bd7592f708af18d" candidate_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042"
I20260809 07:53:21.247524 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd40c7657e34e3b9bd7592f708af18d" candidate_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014dc17a9cbe4665a8571a3626c781f2"
I20260809 07:53:21.247653 32228 raft_consensus.cc:3037] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:21.247653 32436 raft_consensus.cc:3037] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:21.248874 32228 raft_consensus.cc:2454] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ce9b38adfe24c2882b8cfbec30903cb in term 1.
I20260809 07:53:21.249137 32436 raft_consensus.cc:2454] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ce9b38adfe24c2882b8cfbec30903cb in term 1.
I20260809 07:53:21.249184 31897 leader_election.cc:302] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f5f4ab9c0fe4d65b67c36f0cd009042, 7ce9b38adfe24c2882b8cfbec30903cb; no voters:
I20260809 07:53:21.249455 32562 raft_consensus.cc:2781] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:21.249665 32562 raft_consensus.cc:686] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 LEADER]: Becoming Leader. State: Replica: 7ce9b38adfe24c2882b8cfbec30903cb, State: Running, Role: LEADER
I20260809 07:53:21.250002 32562 consensus_queue.cc:237] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.253000 31750 catalog_manager.cc:5495] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ce9b38adfe24c2882b8cfbec30903cb (127.28.112.1). New cstate: current_term: 1 leader_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } health_report { overall_health: UNKNOWN } } }
W20260809 07:53:21.288117 32074 tablet.cc:2369] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 07:53:21 dist-test-slave-vpvm krb5kdc[31680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262000, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:21.343189 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.335605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:46557 (local address 127.0.0.1:48784)
0809 07:53:21.335865 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:21.335878 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:21.335992 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:21.336231 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:21.336234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:21.336650 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:21.336745 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.336752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.337459 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.337463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.338022 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.338030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.338428 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.338432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.338503 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.338991 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:21.339003 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:21.340373 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:21.341800 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:21.341809 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:21.341814 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:21.342089 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:21.342323 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:21.342325 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:21.342326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:21.342408 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:21.342699 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:21.342704 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:21.342814 (+ 110us) client_negotiation.cc:769] Sending connection context
0809 07:53:21.342901 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 07:53:21.343021 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
I20260809 07:53:21.343212 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.335686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.28.112.62:46557)
0809 07:53:21.335808 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:21.335813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:21.335951 (+ 138us) server_negotiation.cc:407] Connection header received
0809 07:53:21.336082 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:21.336085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:21.336125 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:21.336192 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:21.336863 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.337360 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.338137 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.338336 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.338421 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.340499 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:21.340503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:21.340505 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:21.340527 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:21.341705 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:21.342178 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:21.342181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:21.342182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:21.342252 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:21.342464 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:21.342466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:21.342467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:21.342628 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:21.342694 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:21.342959 (+ 265us) server_negotiation.cc:299] Negotiation successful
0809 07:53:21.343079 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":40}
I20260809 07:53:21.347064 31750 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:48784:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 07:53:21.347808 31750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:53:21.352460 32202 tablet_service.cc:1467] Processing CreateTablet for tablet 42e84dde9bfc4c859b6f7c141fa5b86f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1736ff6699714b37adccd1233f22c26f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 07:53:21.352368 31965 tablet_service.cc:1467] Processing CreateTablet for tablet 42e84dde9bfc4c859b6f7c141fa5b86f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1736ff6699714b37adccd1233f22c26f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 07:53:21.352562 32409 tablet_service.cc:1467] Processing CreateTablet for tablet 42e84dde9bfc4c859b6f7c141fa5b86f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1736ff6699714b37adccd1233f22c26f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 07:53:21.353076 32202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e84dde9bfc4c859b6f7c141fa5b86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:21.353091 32409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e84dde9bfc4c859b6f7c141fa5b86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:21.353091 31965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e84dde9bfc4c859b6f7c141fa5b86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:21.355597 32544 tablet_bootstrap.cc:492] T 42e84dde9bfc4c859b6f7c141fa5b86f P 0f5f4ab9c0fe4d65b67c36f0cd009042: Bootstrap starting.
I20260809 07:53:21.355858 32550 tablet_bootstrap.cc:492] T 42e84dde9bfc4c859b6f7c141fa5b86f P 7ce9b38adfe24c2882b8cfbec30903cb: Bootstrap starting.
I20260809 07:53:21.356235 32547 tablet_bootstrap.cc:492] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2: Bootstrap starting.
I20260809 07:53:21.356527 32544 tablet_bootstrap.cc:654] T 42e84dde9bfc4c859b6f7c141fa5b86f P 0f5f4ab9c0fe4d65b67c36f0cd009042: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:21.356721 32550 tablet_bootstrap.cc:654] T 42e84dde9bfc4c859b6f7c141fa5b86f P 7ce9b38adfe24c2882b8cfbec30903cb: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:21.357156 32547 tablet_bootstrap.cc:654] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:21.357826 32544 tablet_bootstrap.cc:492] T 42e84dde9bfc4c859b6f7c141fa5b86f P 0f5f4ab9c0fe4d65b67c36f0cd009042: No bootstrap required, opened a new log
I20260809 07:53:21.357908 32544 ts_tablet_manager.cc:1397] T 42e84dde9bfc4c859b6f7c141fa5b86f P 0f5f4ab9c0fe4d65b67c36f0cd009042: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260809 07:53:21.358223 32550 tablet_bootstrap.cc:492] T 42e84dde9bfc4c859b6f7c141fa5b86f P 7ce9b38adfe24c2882b8cfbec30903cb: No bootstrap required, opened a new log
I20260809 07:53:21.358315 32550 ts_tablet_manager.cc:1397] T 42e84dde9bfc4c859b6f7c141fa5b86f P 7ce9b38adfe24c2882b8cfbec30903cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260809 07:53:21.358319 32544 raft_consensus.cc:348] T 42e84dde9bfc4c859b6f7c141fa5b86f P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.358423 32544 raft_consensus.cc:374] T 42e84dde9bfc4c859b6f7c141fa5b86f P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:21.358446 32544 raft_consensus.cc:729] T 42e84dde9bfc4c859b6f7c141fa5b86f P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5f4ab9c0fe4d65b67c36f0cd009042, State: Initialized, Role: FOLLOWER
I20260809 07:53:21.358554 32544 consensus_queue.cc:260] T 42e84dde9bfc4c859b6f7c141fa5b86f P 0f5f4ab9c0fe4d65b67c36f0cd009042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.358676 32547 tablet_bootstrap.cc:492] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2: No bootstrap required, opened a new log
I20260809 07:53:21.358752 32547 ts_tablet_manager.cc:1397] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 07:53:21.358803 32544 ts_tablet_manager.cc:1428] T 42e84dde9bfc4c859b6f7c141fa5b86f P 0f5f4ab9c0fe4d65b67c36f0cd009042: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 07:53:21.358767 32550 raft_consensus.cc:348] T 42e84dde9bfc4c859b6f7c141fa5b86f P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.358856 32550 raft_consensus.cc:374] T 42e84dde9bfc4c859b6f7c141fa5b86f P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:21.358889 32550 raft_consensus.cc:729] T 42e84dde9bfc4c859b6f7c141fa5b86f P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce9b38adfe24c2882b8cfbec30903cb, State: Initialized, Role: FOLLOWER
I20260809 07:53:21.358845 32558 tablet_replica.cc:439] T 42e84dde9bfc4c859b6f7c141fa5b86f P 0f5f4ab9c0fe4d65b67c36f0cd009042: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } } }
I20260809 07:53:21.358971 32558 tablet_replica.cc:442] T 42e84dde9bfc4c859b6f7c141fa5b86f P 0f5f4ab9c0fe4d65b67c36f0cd009042: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:21.358996 32550 consensus_queue.cc:260] T 42e84dde9bfc4c859b6f7c141fa5b86f P 7ce9b38adfe24c2882b8cfbec30903cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.359175 32547 raft_consensus.cc:348] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.359243 32550 ts_tablet_manager.cc:1428] T 42e84dde9bfc4c859b6f7c141fa5b86f P 7ce9b38adfe24c2882b8cfbec30903cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:21.359266 32547 raft_consensus.cc:374] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:21.359295 32547 raft_consensus.cc:729] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014dc17a9cbe4665a8571a3626c781f2, State: Initialized, Role: FOLLOWER
I20260809 07:53:21.359405 32547 consensus_queue.cc:260] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.359460 32562 tablet_replica.cc:439] T 42e84dde9bfc4c859b6f7c141fa5b86f P 7ce9b38adfe24c2882b8cfbec30903cb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } } }
I20260809 07:53:21.359579 32562 tablet_replica.cc:442] T 42e84dde9bfc4c859b6f7c141fa5b86f P 7ce9b38adfe24c2882b8cfbec30903cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:21.359644 32547 ts_tablet_manager.cc:1428] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:21.359776 32561 tablet_replica.cc:439] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } } }
I20260809 07:53:21.359880 32561 tablet_replica.cc:442] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:21.469218 32561 raft_consensus.cc:482] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:21.469372 32561 raft_consensus.cc:504] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.470595 32561 leader_election.cc:288] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ce9b38adfe24c2882b8cfbec30903cb (127.28.112.1:36585), 0f5f4ab9c0fe4d65b67c36f0cd009042 (127.28.112.2:34431)
I20260809 07:53:21.474761 32538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.470953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:42573 (local address 127.28.112.2:34431)
0809 07:53:21.471128 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:21.471132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:21.471203 (+ 71us) server_negotiation.cc:407] Connection header received
0809 07:53:21.471372 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:21.471375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:21.471416 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:21.471478 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:21.471794 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.472273 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.473495 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.474119 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.474235 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.474288 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:21.474570 (+ 282us) server_negotiation.cc:299] Negotiation successful
0809 07:53:21.474634 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260809 07:53:21.474761 32543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.470925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:49287 (local address 127.28.112.1:36585)
0809 07:53:21.471047 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:21.471053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:21.471225 (+ 172us) server_negotiation.cc:407] Connection header received
0809 07:53:21.471375 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:21.471378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:21.471422 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:21.471488 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:21.471784 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.472281 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.473480 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.474124 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.474235 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.474288 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:21.474573 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 07:53:21.474636 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 07:53:21.474825 32585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.470841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:34431 (local address 127.28.112.3:42573)
0809 07:53:21.471124 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:21.471135 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:21.471269 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:21.471503 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:21.471506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:21.471530 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:21.471688 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.471694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.472377 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.472381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.473312 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.473325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.474232 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.474236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.474411 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.474461 (+ 50us) client_negotiation.cc:769] Sending connection context
0809 07:53:21.474536 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 07:53:21.474642 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":133,"spinlock_wait_cycles":47360,"thread_start_us":70,"threads_started":1}
I20260809 07:53:21.474829 32586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.470823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:36585 (local address 127.28.112.3:49287)
0809 07:53:21.471114 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:21.471130 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:21.471270 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:21.471506 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:21.471509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:21.471533 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:21.471672 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.471677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.472384 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.472388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.473336 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.473347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.474245 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.474249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.474411 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.474462 (+ 51us) client_negotiation.cc:769] Sending connection context
0809 07:53:21.474537 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 07:53:21.474634 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":114,"spinlock_wait_cycles":5760,"thread_start_us":83,"threads_started":1}
I20260809 07:53:21.475440 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e84dde9bfc4c859b6f7c141fa5b86f" candidate_uuid: "014dc17a9cbe4665a8571a3626c781f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" is_pre_election: true
I20260809 07:53:21.475622 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e84dde9bfc4c859b6f7c141fa5b86f" candidate_uuid: "014dc17a9cbe4665a8571a3626c781f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" is_pre_election: true
I20260809 07:53:21.475687 32002 raft_consensus.cc:2454] T 42e84dde9bfc4c859b6f7c141fa5b86f P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014dc17a9cbe4665a8571a3626c781f2 in term 0.
I20260809 07:53:21.475768 32228 raft_consensus.cc:2454] T 42e84dde9bfc4c859b6f7c141fa5b86f P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014dc17a9cbe4665a8571a3626c781f2 in term 0.
I20260809 07:53:21.476155 32342 leader_election.cc:302] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 014dc17a9cbe4665a8571a3626c781f2, 0f5f4ab9c0fe4d65b67c36f0cd009042, 7ce9b38adfe24c2882b8cfbec30903cb; no voters:
I20260809 07:53:21.476433 32561 raft_consensus.cc:2781] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:53:21.476493 32561 raft_consensus.cc:482] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:53:21.476519 32561 raft_consensus.cc:3037] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:21.477414 32561 raft_consensus.cc:504] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.477730 32561 leader_election.cc:288] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 [CANDIDATE]: Term 1 election: Requested vote from peers 7ce9b38adfe24c2882b8cfbec30903cb (127.28.112.1:36585), 0f5f4ab9c0fe4d65b67c36f0cd009042 (127.28.112.2:34431)
I20260809 07:53:21.478084 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e84dde9bfc4c859b6f7c141fa5b86f" candidate_uuid: "014dc17a9cbe4665a8571a3626c781f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042"
I20260809 07:53:21.478183 32228 raft_consensus.cc:3037] T 42e84dde9bfc4c859b6f7c141fa5b86f P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:21.479004 32228 raft_consensus.cc:2454] T 42e84dde9bfc4c859b6f7c141fa5b86f P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014dc17a9cbe4665a8571a3626c781f2 in term 1.
I20260809 07:53:21.479228 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e84dde9bfc4c859b6f7c141fa5b86f" candidate_uuid: "014dc17a9cbe4665a8571a3626c781f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce9b38adfe24c2882b8cfbec30903cb"
I20260809 07:53:21.479261 32342 leader_election.cc:302] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014dc17a9cbe4665a8571a3626c781f2, 0f5f4ab9c0fe4d65b67c36f0cd009042; no voters:
I20260809 07:53:21.479346 32002 raft_consensus.cc:3037] T 42e84dde9bfc4c859b6f7c141fa5b86f P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:21.479413 32561 raft_consensus.cc:2781] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:21.479594 32561 raft_consensus.cc:686] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 [term 1 LEADER]: Becoming Leader. State: Replica: 014dc17a9cbe4665a8571a3626c781f2, State: Running, Role: LEADER
I20260809 07:53:21.479912 32561 consensus_queue.cc:237] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.480273 32002 raft_consensus.cc:2454] T 42e84dde9bfc4c859b6f7c141fa5b86f P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014dc17a9cbe4665a8571a3626c781f2 in term 1.
I20260809 07:53:21.481855 32587 tablet_replica.cc:439] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2: TxnStatusTablet state changed. Reason: New leader 014dc17a9cbe4665a8571a3626c781f2. Latest consensus state: current_term: 1 leader_uuid: "014dc17a9cbe4665a8571a3626c781f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } } }
I20260809 07:53:21.481982 32587 tablet_replica.cc:442] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2: This TxnStatusTablet replica's current role is: LEADER
I20260809 07:53:21.482295 32590 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 07:53:21.482501 32590 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 07:53:21.482684 31751 catalog_manager.cc:5495] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 014dc17a9cbe4665a8571a3626c781f2 (127.28.112.3). New cstate: current_term: 1 leader_uuid: "014dc17a9cbe4665a8571a3626c781f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } health_report { overall_health: HEALTHY } } }
I20260809 07:53:21.515681 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.511814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35110 (local address 127.28.112.3:36449)
0809 07:53:21.511922 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:21.511926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:21.511940 (+ 14us) server_negotiation.cc:407] Connection header received
0809 07:53:21.512159 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:21.512162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:21.512217 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:21.512291 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:21.512522 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.512999 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.514203 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.514412 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.514488 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.514824 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:21.515324 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:21.515414 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:21.515511 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 07:53:21.515560 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 07:53:21.515729 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.511668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:36449 (local address 127.0.0.1:35110)
0809 07:53:21.511862 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:21.511877 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:21.511984 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:21.512312 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:21.512315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:21.512323 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 07:53:21.512433 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.512438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.513087 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.513091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.514099 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.514111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.514508 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.514512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.514602 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.514659 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 07:53:21.515410 (+ 751us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 07:53:21.515425 (+ 15us) client_negotiation.cc:769] Sending connection context
0809 07:53:21.515502 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 07:53:21.515585 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260809 07:53:21.521359 32228 raft_consensus.cc:1261] T 42e84dde9bfc4c859b6f7c141fa5b86f P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 1 FOLLOWER]: Refusing update from remote peer 014dc17a9cbe4665a8571a3626c781f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:21.521358 32002 raft_consensus.cc:1261] T 42e84dde9bfc4c859b6f7c141fa5b86f P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 FOLLOWER]: Refusing update from remote peer 014dc17a9cbe4665a8571a3626c781f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:21.522038 32587 consensus_queue.cc:1035] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:21.522240 32561 consensus_queue.cc:1035] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:21.525336 32562 tablet_replica.cc:439] T 42e84dde9bfc4c859b6f7c141fa5b86f P 7ce9b38adfe24c2882b8cfbec30903cb: TxnStatusTablet state changed. Reason: New leader 014dc17a9cbe4665a8571a3626c781f2. Latest consensus state: current_term: 1 leader_uuid: "014dc17a9cbe4665a8571a3626c781f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } } }
I20260809 07:53:21.525458 32562 tablet_replica.cc:442] T 42e84dde9bfc4c859b6f7c141fa5b86f P 7ce9b38adfe24c2882b8cfbec30903cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:21.525627 32558 tablet_replica.cc:439] T 42e84dde9bfc4c859b6f7c141fa5b86f P 0f5f4ab9c0fe4d65b67c36f0cd009042: TxnStatusTablet state changed. Reason: New leader 014dc17a9cbe4665a8571a3626c781f2. Latest consensus state: current_term: 1 leader_uuid: "014dc17a9cbe4665a8571a3626c781f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } } }
I20260809 07:53:21.525729 32558 tablet_replica.cc:442] T 42e84dde9bfc4c859b6f7c141fa5b86f P 0f5f4ab9c0fe4d65b67c36f0cd009042: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:21.530066 32562 tablet_replica.cc:439] T 42e84dde9bfc4c859b6f7c141fa5b86f P 7ce9b38adfe24c2882b8cfbec30903cb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "014dc17a9cbe4665a8571a3626c781f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } } }
I20260809 07:53:21.530191 32562 tablet_replica.cc:442] T 42e84dde9bfc4c859b6f7c141fa5b86f P 7ce9b38adfe24c2882b8cfbec30903cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:21.530395 32595 mvcc.cc:205] Tried to move back new op lower bound from 7316529158224609280 to 7316529158064652288. Current Snapshot: MvccSnapshot[applied={T|T < 7316529158224609280}]
I20260809 07:53:21.530576 32558 tablet_replica.cc:439] T 42e84dde9bfc4c859b6f7c141fa5b86f P 0f5f4ab9c0fe4d65b67c36f0cd009042: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "014dc17a9cbe4665a8571a3626c781f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } } }
I20260809 07:53:21.530670 32558 tablet_replica.cc:442] T 42e84dde9bfc4c859b6f7c141fa5b86f P 0f5f4ab9c0fe4d65b67c36f0cd009042: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:21.532439 32587 tablet_replica.cc:439] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "014dc17a9cbe4665a8571a3626c781f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } } }
I20260809 07:53:21.532550 32587 tablet_replica.cc:442] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2: This TxnStatusTablet replica's current role is: LEADER
I20260809 07:53:21.533290 32598 tablet_replica.cc:439] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "014dc17a9cbe4665a8571a3626c781f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } } }
I20260809 07:53:21.533398 32598 tablet_replica.cc:442] T 42e84dde9bfc4c859b6f7c141fa5b86f P 014dc17a9cbe4665a8571a3626c781f2: This TxnStatusTablet replica's current role is: LEADER
I20260809 07:53:21.547967 32543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.544321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53384 (local address 127.28.112.1:36585)
0809 07:53:21.544442 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:21.544446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:21.544688 (+ 242us) server_negotiation.cc:407] Connection header received
0809 07:53:21.544834 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:21.544838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:21.544890 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:21.544975 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:21.545215 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.545713 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.546614 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.546830 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.546917 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.547183 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:21.547590 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:21.547693 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:21.547779 (+ 86us) server_negotiation.cc:299] Negotiation successful
0809 07:53:21.547831 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 07:53:21.558396 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.555074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:35671 (local address 127.28.112.3:36449)
0809 07:53:21.555241 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:21.555245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:21.555317 (+ 72us) server_negotiation.cc:407] Connection header received
0809 07:53:21.555442 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:21.555444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:21.555488 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:21.555553 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:21.555780 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.556263 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.557409 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.557607 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.557673 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.558023 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:21.558091 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:21.558142 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:21.558242 (+ 100us) server_negotiation.cc:299] Negotiation successful
0809 07:53:21.558287 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 07:53:21.558457 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.554937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:36449 (local address 127.28.112.1:35671)
0809 07:53:21.555231 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:21.555247 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:21.555357 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:21.555577 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:21.555580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:21.555588 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 07:53:21.555695 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.555701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.556346 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.556349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.557314 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.557325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.557697 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.557701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.557786 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.557837 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 07:53:21.558159 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 07:53:21.558165 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 07:53:21.558223 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 07:53:21.558298 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":102,"threads_started":1}
I20260809 07:53:21.565358 32228 raft_consensus.cc:1261] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 1 FOLLOWER]: Refusing update from remote peer 7ce9b38adfe24c2882b8cfbec30903cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:21.565471 32436 raft_consensus.cc:1261] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2 [term 1 FOLLOWER]: Refusing update from remote peer 7ce9b38adfe24c2882b8cfbec30903cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:21.565804 32567 consensus_queue.cc:1035] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:21.565996 32562 consensus_queue.cc:1035] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:21.569679 32594 mvcc.cc:205] Tried to move back new op lower bound from 7316529158407335936 to 7316529157122523136. Current Snapshot: MvccSnapshot[applied={T|T < 7316529158407335936}]
I20260809 07:53:21.573207 32593 mvcc.cc:205] Tried to move back new op lower bound from 7316529158407335936 to 7316529157122523136. Current Snapshot: MvccSnapshot[applied={T|T < 7316529158407335936}]
I20260809 07:53:21.588495 32543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.585067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:49289 (local address 127.28.112.1:36585)
0809 07:53:21.585191 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:21.585196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:21.585358 (+ 162us) server_negotiation.cc:407] Connection header received
0809 07:53:21.585493 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:21.585495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:21.585533 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:21.585622 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:21.585854 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.586427 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.587609 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.587816 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.587891 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.588169 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:21.588230 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:21.588276 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:21.588360 (+ 84us) server_negotiation.cc:299] Negotiation successful
0809 07:53:21.588399 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 07:53:21.588579 32615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:21.584960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:36585 (local address 127.28.112.3:49289)
0809 07:53:21.585274 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:21.585290 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:21.585400 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:21.585644 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:21.585647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:21.585655 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 07:53:21.585758 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.585765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.586537 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.586541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.587496 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:21.587508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:21.587923 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:21.587928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:21.588010 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:21.588060 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 07:53:21.588298 (+ 238us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 07:53:21.588303 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 07:53:21.588357 (+ 54us) client_negotiation.cc:240] Negotiation successful
0809 07:53:21.588440 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":101,"threads_started":1}
I20260809 07:53:21.610026 31751 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:21.610195 31751 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:21.612373 31751 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14: Sending DeleteTablet for 3 replicas of tablet ebd40c7657e34e3b9bd7592f708af18d
I20260809 07:53:21.613080 32202 tablet_service.cc:1514] Processing DeleteTablet for tablet ebd40c7657e34e3b9bd7592f708af18d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:21 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:57090
I20260809 07:53:21.613193 32409 tablet_service.cc:1514] Processing DeleteTablet for tablet ebd40c7657e34e3b9bd7592f708af18d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:21 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:35100
I20260809 07:53:21.613579 32623 tablet_replica.cc:330] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042: stopping tablet replica
I20260809 07:53:21.613917 32623 raft_consensus.cc:2229] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:21.614253 32623 raft_consensus.cc:2258] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:21.614368 31751 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:53:21.614775 31751 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:53:21.615784 32623 ts_tablet_manager.cc:1905] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:21.618978 32623 ts_tablet_manager.cc:1918] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 07:53:21.619156 32623 log.cc:1196] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/ebd40c7657e34e3b9bd7592f708af18d
I20260809 07:53:21.619570 32623 ts_tablet_manager.cc:1939] T ebd40c7657e34e3b9bd7592f708af18d P 0f5f4ab9c0fe4d65b67c36f0cd009042: Deleting consensus metadata
I20260809 07:53:21.620038 32409 tablet_service.cc:1467] Processing CreateTablet for tablet 018181c72b5a4e079041d4ca1a023e11 (DEFAULT_TABLE table=test-table [id=3e52690bee9245748df69f2d6ead3734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:21.620332 32409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 018181c72b5a4e079041d4ca1a023e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:21.622124 32624 tablet_replica.cc:330] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2: stopping tablet replica
I20260809 07:53:21.622359 32547 tablet_bootstrap.cc:492] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2: Bootstrap starting.
I20260809 07:53:21.622500 32624 raft_consensus.cc:2229] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:21.622575 31724 catalog_manager.cc:4841] TS 0f5f4ab9c0fe4d65b67c36f0cd009042 (127.28.112.2:34431): tablet ebd40c7657e34e3b9bd7592f708af18d (table test-table [id=8939059299ec4764bbfc5ad17fc3ecdb]) successfully deleted
I20260809 07:53:21.622658 32624 raft_consensus.cc:2258] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:21.623334 32547 tablet_bootstrap.cc:654] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:21.623425 31964 tablet_service.cc:1467] Processing CreateTablet for tablet 018181c72b5a4e079041d4ca1a023e11 (DEFAULT_TABLE table=test-table [id=3e52690bee9245748df69f2d6ead3734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:21.623724 31964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 018181c72b5a4e079041d4ca1a023e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:21.624223 31965 tablet_service.cc:1514] Processing DeleteTablet for tablet ebd40c7657e34e3b9bd7592f708af18d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:21 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:53368
I20260809 07:53:21.624668 32626 tablet_replica.cc:330] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb: stopping tablet replica
I20260809 07:53:21.624763 32202 tablet_service.cc:1467] Processing CreateTablet for tablet 018181c72b5a4e079041d4ca1a023e11 (DEFAULT_TABLE table=test-table [id=3e52690bee9245748df69f2d6ead3734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:21.624998 32202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 018181c72b5a4e079041d4ca1a023e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:21.624998 32626 raft_consensus.cc:2229] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:53:21.625363 32626 raft_consensus.cc:2258] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:21.626183 32550 tablet_bootstrap.cc:492] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb: Bootstrap starting.
I20260809 07:53:21.626595 32547 tablet_bootstrap.cc:492] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2: No bootstrap required, opened a new log
I20260809 07:53:21.626672 32547 ts_tablet_manager.cc:1397] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260809 07:53:21.626662 32626 ts_tablet_manager.cc:1905] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:21.627097 32550 tablet_bootstrap.cc:654] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:21.627097 32547 raft_consensus.cc:348] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.627221 32547 raft_consensus.cc:374] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:21.627254 32547 raft_consensus.cc:729] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014dc17a9cbe4665a8571a3626c781f2, State: Initialized, Role: FOLLOWER
I20260809 07:53:21.627377 32547 consensus_queue.cc:260] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.627616 32547 ts_tablet_manager.cc:1428] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:21.628414 32624 ts_tablet_manager.cc:1905] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:21.629277 32626 ts_tablet_manager.cc:1918] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 07:53:21.629413 32626 log.cc:1196] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/wal/wals/ebd40c7657e34e3b9bd7592f708af18d
I20260809 07:53:21.629801 32626 ts_tablet_manager.cc:1939] T ebd40c7657e34e3b9bd7592f708af18d P 7ce9b38adfe24c2882b8cfbec30903cb: Deleting consensus metadata
I20260809 07:53:21.630927 31723 catalog_manager.cc:4841] TS 7ce9b38adfe24c2882b8cfbec30903cb (127.28.112.1:36585): tablet ebd40c7657e34e3b9bd7592f708af18d (table test-table [id=8939059299ec4764bbfc5ad17fc3ecdb]) successfully deleted
I20260809 07:53:21.631235 32624 ts_tablet_manager.cc:1918] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 07:53:21.631323 32624 log.cc:1196] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/ebd40c7657e34e3b9bd7592f708af18d
I20260809 07:53:21.631625 32624 ts_tablet_manager.cc:1939] T ebd40c7657e34e3b9bd7592f708af18d P 014dc17a9cbe4665a8571a3626c781f2: Deleting consensus metadata
I20260809 07:53:21.632403 32544 tablet_bootstrap.cc:492] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042: Bootstrap starting.
I20260809 07:53:21.632671 31722 catalog_manager.cc:4841] TS 014dc17a9cbe4665a8571a3626c781f2 (127.28.112.3:36449): tablet ebd40c7657e34e3b9bd7592f708af18d (table test-table [id=8939059299ec4764bbfc5ad17fc3ecdb]) successfully deleted
I20260809 07:53:21.633419 32544 tablet_bootstrap.cc:654] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:21.634568 32544 tablet_bootstrap.cc:492] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042: No bootstrap required, opened a new log
I20260809 07:53:21.634647 32544 ts_tablet_manager.cc:1397] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:21.635003 32544 raft_consensus.cc:348] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.635097 32544 raft_consensus.cc:374] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:21.635128 32544 raft_consensus.cc:729] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5f4ab9c0fe4d65b67c36f0cd009042, State: Initialized, Role: FOLLOWER
I20260809 07:53:21.635250 32544 consensus_queue.cc:260] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.635483 32544 ts_tablet_manager.cc:1428] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:21.638631 32550 tablet_bootstrap.cc:492] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb: No bootstrap required, opened a new log
I20260809 07:53:21.638713 32550 ts_tablet_manager.cc:1397] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260809 07:53:21.639079 32550 raft_consensus.cc:348] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.639169 32550 raft_consensus.cc:374] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:21.639199 32550 raft_consensus.cc:729] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce9b38adfe24c2882b8cfbec30903cb, State: Initialized, Role: FOLLOWER
I20260809 07:53:21.639320 32550 consensus_queue.cc:260] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.639580 32550 ts_tablet_manager.cc:1428] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:21.807907 32567 raft_consensus.cc:482] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:21.808043 32567 raft_consensus.cc:504] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.808419 32567 leader_election.cc:288] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f5f4ab9c0fe4d65b67c36f0cd009042 (127.28.112.2:34431), 014dc17a9cbe4665a8571a3626c781f2 (127.28.112.3:36449)
I20260809 07:53:21.808900 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018181c72b5a4e079041d4ca1a023e11" candidate_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" is_pre_election: true
I20260809 07:53:21.808900 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018181c72b5a4e079041d4ca1a023e11" candidate_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014dc17a9cbe4665a8571a3626c781f2" is_pre_election: true
I20260809 07:53:21.809063 32436 raft_consensus.cc:2454] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ce9b38adfe24c2882b8cfbec30903cb in term 0.
I20260809 07:53:21.809063 32228 raft_consensus.cc:2454] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ce9b38adfe24c2882b8cfbec30903cb in term 0.
I20260809 07:53:21.809382 31895 leader_election.cc:302] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 014dc17a9cbe4665a8571a3626c781f2, 0f5f4ab9c0fe4d65b67c36f0cd009042, 7ce9b38adfe24c2882b8cfbec30903cb; no voters:
I20260809 07:53:21.809571 32567 raft_consensus.cc:2781] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:53:21.809630 32567 raft_consensus.cc:482] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:53:21.809669 32567 raft_consensus.cc:3037] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:21.810703 32567 raft_consensus.cc:504] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.811008 32567 leader_election.cc:288] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [CANDIDATE]: Term 1 election: Requested vote from peers 0f5f4ab9c0fe4d65b67c36f0cd009042 (127.28.112.2:34431), 014dc17a9cbe4665a8571a3626c781f2 (127.28.112.3:36449)
I20260809 07:53:21.811348 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018181c72b5a4e079041d4ca1a023e11" candidate_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042"
I20260809 07:53:21.811348 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018181c72b5a4e079041d4ca1a023e11" candidate_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014dc17a9cbe4665a8571a3626c781f2"
I20260809 07:53:21.811475 32228 raft_consensus.cc:3037] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:21.811475 32436 raft_consensus.cc:3037] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:21.812523 32228 raft_consensus.cc:2454] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ce9b38adfe24c2882b8cfbec30903cb in term 1.
I20260809 07:53:21.812523 32436 raft_consensus.cc:2454] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ce9b38adfe24c2882b8cfbec30903cb in term 1.
I20260809 07:53:21.812808 31895 leader_election.cc:302] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 014dc17a9cbe4665a8571a3626c781f2, 0f5f4ab9c0fe4d65b67c36f0cd009042, 7ce9b38adfe24c2882b8cfbec30903cb; no voters:
I20260809 07:53:21.812973 32567 raft_consensus.cc:2781] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:21.813060 32567 raft_consensus.cc:686] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 LEADER]: Becoming Leader. State: Replica: 7ce9b38adfe24c2882b8cfbec30903cb, State: Running, Role: LEADER
I20260809 07:53:21.813205 32567 consensus_queue.cc:237] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:21.814817 31751 catalog_manager.cc:5495] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ce9b38adfe24c2882b8cfbec30903cb (127.28.112.1). New cstate: current_term: 1 leader_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } health_report { overall_health: UNKNOWN } } }
I20260809 07:53:21.835067 32436 raft_consensus.cc:1261] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2 [term 1 FOLLOWER]: Refusing update from remote peer 7ce9b38adfe24c2882b8cfbec30903cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:21.835145 32228 raft_consensus.cc:1261] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 1 FOLLOWER]: Refusing update from remote peer 7ce9b38adfe24c2882b8cfbec30903cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:21.835449 32567 consensus_queue.cc:1035] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:21.835606 32618 consensus_queue.cc:1035] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 07:53:21.848157 32594 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 07:53:23.397058 31751 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:23.397207 31751 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:23.399390 31751 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14: Sending DeleteTablet for 3 replicas of tablet 018181c72b5a4e079041d4ca1a023e11
I20260809 07:53:23.399945 32409 tablet_service.cc:1514] Processing DeleteTablet for tablet 018181c72b5a4e079041d4ca1a023e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:23 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:35100
I20260809 07:53:23.399979 31964 tablet_service.cc:1514] Processing DeleteTablet for tablet 018181c72b5a4e079041d4ca1a023e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:23 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:53368
I20260809 07:53:23.400274 32202 tablet_service.cc:1514] Processing DeleteTablet for tablet 018181c72b5a4e079041d4ca1a023e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:23 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:57090
I20260809 07:53:23.400596 32677 tablet_replica.cc:330] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042: stopping tablet replica
I20260809 07:53:23.400715 32675 tablet_replica.cc:330] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb: stopping tablet replica
I20260809 07:53:23.400288 32673 tablet_replica.cc:330] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2: stopping tablet replica
I20260809 07:53:23.400760 32677 raft_consensus.cc:2229] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:23.400880 32673 raft_consensus.cc:2229] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:23.400913 32675 raft_consensus.cc:2229] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:53:23.400949 32677 raft_consensus.cc:2258] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:23.401043 32673 raft_consensus.cc:2258] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:23.401176 32675 raft_consensus.cc:2258] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:23.401564 32677 ts_tablet_manager.cc:1905] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:23.401696 32673 ts_tablet_manager.cc:1905] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:23.401774 32675 ts_tablet_manager.cc:1905] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:23.404333 32673 ts_tablet_manager.cc:1918] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 07:53:23.404333 32675 ts_tablet_manager.cc:1918] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 07:53:23.404381 32677 ts_tablet_manager.cc:1918] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 07:53:23.404409 32675 log.cc:1196] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/wal/wals/018181c72b5a4e079041d4ca1a023e11
I20260809 07:53:23.404492 32677 log.cc:1196] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/018181c72b5a4e079041d4ca1a023e11
I20260809 07:53:23.404520 32673 log.cc:1196] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/018181c72b5a4e079041d4ca1a023e11
I20260809 07:53:23.404698 32675 ts_tablet_manager.cc:1939] T 018181c72b5a4e079041d4ca1a023e11 P 7ce9b38adfe24c2882b8cfbec30903cb: Deleting consensus metadata
I20260809 07:53:23.404785 32673 ts_tablet_manager.cc:1939] T 018181c72b5a4e079041d4ca1a023e11 P 014dc17a9cbe4665a8571a3626c781f2: Deleting consensus metadata
I20260809 07:53:23.404874 32677 ts_tablet_manager.cc:1939] T 018181c72b5a4e079041d4ca1a023e11 P 0f5f4ab9c0fe4d65b67c36f0cd009042: Deleting consensus metadata
I20260809 07:53:23.405381 31723 catalog_manager.cc:4841] TS 7ce9b38adfe24c2882b8cfbec30903cb (127.28.112.1:36585): tablet 018181c72b5a4e079041d4ca1a023e11 (table test-table [id=3e52690bee9245748df69f2d6ead3734]) successfully deleted
I20260809 07:53:23.405440 31722 catalog_manager.cc:4841] TS 014dc17a9cbe4665a8571a3626c781f2 (127.28.112.3:36449): tablet 018181c72b5a4e079041d4ca1a023e11 (table test-table [id=3e52690bee9245748df69f2d6ead3734]) successfully deleted
I20260809 07:53:23.405613 31724 catalog_manager.cc:4841] TS 0f5f4ab9c0fe4d65b67c36f0cd009042 (127.28.112.2:34431): tablet 018181c72b5a4e079041d4ca1a023e11 (table test-table [id=3e52690bee9245748df69f2d6ead3734]) successfully deleted
Aug 09 07:53:23 dist-test-slave-vpvm krb5kdc[31680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262003, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 07:53:23 dist-test-slave-vpvm krb5kdc[31680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262003, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:23.430356 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:23.419815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48796 (local address 127.28.112.62:46557)
0809 07:53:23.420096 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:23.420103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:23.420117 (+ 14us) server_negotiation.cc:407] Connection header received
0809 07:53:23.420214 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:23.420217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:23.420261 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:23.420336 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:23.420928 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:23.421455 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:23.422242 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:23.422454 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:23.422527 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:23.427686 (+ 5159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:23.427690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:23.427692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:23.427719 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:23.428954 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:23.429302 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:23.429305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:23.429306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:23.429377 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:23.429562 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:23.429564 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:23.429566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:23.429760 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:23.429858 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:23.430080 (+ 222us) server_negotiation.cc:299] Negotiation successful
0809 07:53:23.430184 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":101,"threads_started":1}
I20260809 07:53:23.432745 31751 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:53:23.433187 31751 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:53:23.437760 31964 tablet_service.cc:1467] Processing CreateTablet for tablet 9a20c895b399495aa900df627af66480 (DEFAULT_TABLE table=test-table [id=592a14e628604f9cb3dd4c45dd49a9a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:23.437860 32202 tablet_service.cc:1467] Processing CreateTablet for tablet 9a20c895b399495aa900df627af66480 (DEFAULT_TABLE table=test-table [id=592a14e628604f9cb3dd4c45dd49a9a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:23.437901 32409 tablet_service.cc:1467] Processing CreateTablet for tablet 9a20c895b399495aa900df627af66480 (DEFAULT_TABLE table=test-table [id=592a14e628604f9cb3dd4c45dd49a9a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:23.438144 32202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a20c895b399495aa900df627af66480. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:23.438130 31964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a20c895b399495aa900df627af66480. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:23.438194 32409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a20c895b399495aa900df627af66480. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:23.440152 32690 tablet_bootstrap.cc:492] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2: Bootstrap starting.
I20260809 07:53:23.440750 32693 tablet_bootstrap.cc:492] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb: Bootstrap starting.
I20260809 07:53:23.441124 32690 tablet_bootstrap.cc:654] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:23.441556 32693 tablet_bootstrap.cc:654] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:23.441674 32692 tablet_bootstrap.cc:492] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042: Bootstrap starting.
I20260809 07:53:23.442426 32690 tablet_bootstrap.cc:492] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2: No bootstrap required, opened a new log
I20260809 07:53:23.442519 32690 ts_tablet_manager.cc:1397] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:23.442531 32692 tablet_bootstrap.cc:654] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:23.442780 32693 tablet_bootstrap.cc:492] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb: No bootstrap required, opened a new log
I20260809 07:53:23.442871 32693 ts_tablet_manager.cc:1397] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:23.443290 32693 raft_consensus.cc:348] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:23.443092 32690 raft_consensus.cc:348] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:23.443495 32690 raft_consensus.cc:374] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:23.443497 32693 raft_consensus.cc:374] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:23.443558 32693 raft_consensus.cc:729] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce9b38adfe24c2882b8cfbec30903cb, State: Initialized, Role: FOLLOWER
I20260809 07:53:23.443603 32690 raft_consensus.cc:729] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014dc17a9cbe4665a8571a3626c781f2, State: Initialized, Role: FOLLOWER
I20260809 07:53:23.443734 32692 tablet_bootstrap.cc:492] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042: No bootstrap required, opened a new log
I20260809 07:53:23.443817 32692 ts_tablet_manager.cc:1397] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 07:53:23.443770 32693 consensus_queue.cc:260] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:23.443785 32690 consensus_queue.cc:260] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:23.444029 32693 ts_tablet_manager.cc:1428] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:23.444239 32692 raft_consensus.cc:348] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:23.444343 32692 raft_consensus.cc:374] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:23.444375 32692 raft_consensus.cc:729] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5f4ab9c0fe4d65b67c36f0cd009042, State: Initialized, Role: FOLLOWER
I20260809 07:53:23.444499 32690 ts_tablet_manager.cc:1428] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:23.444487 32692 consensus_queue.cc:260] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:23.444851 32692 ts_tablet_manager.cc:1428] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:23.619475 32666 raft_consensus.cc:482] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:23.619642 32666 raft_consensus.cc:504] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:23.620024 32666 leader_election.cc:288] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f5f4ab9c0fe4d65b67c36f0cd009042 (127.28.112.2:34431), 014dc17a9cbe4665a8571a3626c781f2 (127.28.112.3:36449)
I20260809 07:53:23.620504 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a20c895b399495aa900df627af66480" candidate_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014dc17a9cbe4665a8571a3626c781f2" is_pre_election: true
I20260809 07:53:23.620507 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a20c895b399495aa900df627af66480" candidate_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" is_pre_election: true
I20260809 07:53:23.620647 32436 raft_consensus.cc:2454] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ce9b38adfe24c2882b8cfbec30903cb in term 0.
I20260809 07:53:23.620659 32228 raft_consensus.cc:2454] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ce9b38adfe24c2882b8cfbec30903cb in term 0.
I20260809 07:53:23.620966 31897 leader_election.cc:302] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f5f4ab9c0fe4d65b67c36f0cd009042, 7ce9b38adfe24c2882b8cfbec30903cb; no voters:
I20260809 07:53:23.621140 32666 raft_consensus.cc:2781] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:53:23.621204 32666 raft_consensus.cc:482] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:53:23.621249 32666 raft_consensus.cc:3037] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:23.622284 32666 raft_consensus.cc:504] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:23.622592 32666 leader_election.cc:288] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [CANDIDATE]: Term 1 election: Requested vote from peers 0f5f4ab9c0fe4d65b67c36f0cd009042 (127.28.112.2:34431), 014dc17a9cbe4665a8571a3626c781f2 (127.28.112.3:36449)
I20260809 07:53:23.622905 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a20c895b399495aa900df627af66480" candidate_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042"
I20260809 07:53:23.622932 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a20c895b399495aa900df627af66480" candidate_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014dc17a9cbe4665a8571a3626c781f2"
I20260809 07:53:23.623004 32228 raft_consensus.cc:3037] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:23.623062 32436 raft_consensus.cc:3037] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:23.623867 32228 raft_consensus.cc:2454] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ce9b38adfe24c2882b8cfbec30903cb in term 1.
I20260809 07:53:23.623974 32436 raft_consensus.cc:2454] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ce9b38adfe24c2882b8cfbec30903cb in term 1.
I20260809 07:53:23.624128 31897 leader_election.cc:302] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f5f4ab9c0fe4d65b67c36f0cd009042, 7ce9b38adfe24c2882b8cfbec30903cb; no voters:
I20260809 07:53:23.624303 32666 raft_consensus.cc:2781] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:23.624408 32666 raft_consensus.cc:686] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 LEADER]: Becoming Leader. State: Replica: 7ce9b38adfe24c2882b8cfbec30903cb, State: Running, Role: LEADER
I20260809 07:53:23.624554 32666 consensus_queue.cc:237] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:23.626225 31751 catalog_manager.cc:5495] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ce9b38adfe24c2882b8cfbec30903cb (127.28.112.1). New cstate: current_term: 1 leader_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } health_report { overall_health: UNKNOWN } } }
I20260809 07:53:23.682884 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:23.680121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53394 (local address 127.28.112.1:36585)
0809 07:53:23.680373 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:23.680379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:23.680391 (+ 12us) server_negotiation.cc:407] Connection header received
0809 07:53:23.680430 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:23.680432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:23.680473 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:23.680542 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:23.680756 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:23.681270 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:23.682036 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:23.682253 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:23.682320 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:23.682523 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:23.682587 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:23.682640 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:23.682732 (+ 92us) server_negotiation.cc:299] Negotiation successful
0809 07:53:23.682773 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
I20260809 07:53:23.688217 32228 raft_consensus.cc:1261] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 1 FOLLOWER]: Refusing update from remote peer 7ce9b38adfe24c2882b8cfbec30903cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:23.688313 32436 raft_consensus.cc:1261] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2 [term 1 FOLLOWER]: Refusing update from remote peer 7ce9b38adfe24c2882b8cfbec30903cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:23.688587 32666 consensus_queue.cc:1035] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:23.688709 32637 consensus_queue.cc:1035] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:23.715688 31752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:23.715816 31752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:23.717543 31752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14: Sending DeleteTablet for 3 replicas of tablet 9a20c895b399495aa900df627af66480
I20260809 07:53:23.717999 31964 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a20c895b399495aa900df627af66480 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:23 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:53368
I20260809 07:53:23.718191 32675 tablet_replica.cc:330] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb: stopping tablet replica
I20260809 07:53:23.718163 32202 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a20c895b399495aa900df627af66480 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:23 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:57090
I20260809 07:53:23.718181 32409 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a20c895b399495aa900df627af66480 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:23 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:35100
I20260809 07:53:23.718297 32675 raft_consensus.cc:2229] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:53:23.718343 32673 tablet_replica.cc:330] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2: stopping tablet replica
I20260809 07:53:23.718425 32673 raft_consensus.cc:2229] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:23.718461 32675 raft_consensus.cc:2258] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:23.718508 32673 raft_consensus.cc:2258] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:23.718509 32677 tablet_replica.cc:330] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042: stopping tablet replica
I20260809 07:53:23.718636 32677 raft_consensus.cc:2229] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:23.718778 32677 raft_consensus.cc:2258] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:23.719167 32673 ts_tablet_manager.cc:1905] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:23.719197 32675 ts_tablet_manager.cc:1905] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:23.721095 32675 ts_tablet_manager.cc:1918] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 07:53:23.721167 32675 log.cc:1196] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-0/wal/wals/9a20c895b399495aa900df627af66480
I20260809 07:53:23.721158 32673 ts_tablet_manager.cc:1918] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 07:53:23.721220 32673 log.cc:1196] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/9a20c895b399495aa900df627af66480
I20260809 07:53:23.721410 32675 ts_tablet_manager.cc:1939] T 9a20c895b399495aa900df627af66480 P 7ce9b38adfe24c2882b8cfbec30903cb: Deleting consensus metadata
I20260809 07:53:23.721451 32673 ts_tablet_manager.cc:1939] T 9a20c895b399495aa900df627af66480 P 014dc17a9cbe4665a8571a3626c781f2: Deleting consensus metadata
I20260809 07:53:23.722134 31722 catalog_manager.cc:4841] TS 014dc17a9cbe4665a8571a3626c781f2 (127.28.112.3:36449): tablet 9a20c895b399495aa900df627af66480 (table test-table [id=592a14e628604f9cb3dd4c45dd49a9a0]) successfully deleted
I20260809 07:53:23.722146 32677 ts_tablet_manager.cc:1905] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:23.722194 31752 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:53:23.722591 31752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:53:23.724424 32677 ts_tablet_manager.cc:1918] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 07:53:23.724496 32677 log.cc:1196] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/9a20c895b399495aa900df627af66480
I20260809 07:53:23.724743 32677 ts_tablet_manager.cc:1939] T 9a20c895b399495aa900df627af66480 P 0f5f4ab9c0fe4d65b67c36f0cd009042: Deleting consensus metadata
I20260809 07:53:23.725328 31724 catalog_manager.cc:4841] TS 0f5f4ab9c0fe4d65b67c36f0cd009042 (127.28.112.2:34431): tablet 9a20c895b399495aa900df627af66480 (table test-table [id=592a14e628604f9cb3dd4c45dd49a9a0]) successfully deleted
I20260809 07:53:23.725592 31723 catalog_manager.cc:4841] TS 7ce9b38adfe24c2882b8cfbec30903cb (127.28.112.1:36585): tablet 9a20c895b399495aa900df627af66480 (table test-table [id=592a14e628604f9cb3dd4c45dd49a9a0]) successfully deleted
I20260809 07:53:23.726886 32202 tablet_service.cc:1467] Processing CreateTablet for tablet df82d6c0d06e4b458abc770716d7a62e (DEFAULT_TABLE table=test-table [id=3ea0d944dee84880af3e763f99971e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:23.726886 31964 tablet_service.cc:1467] Processing CreateTablet for tablet df82d6c0d06e4b458abc770716d7a62e (DEFAULT_TABLE table=test-table [id=3ea0d944dee84880af3e763f99971e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:23.727119 31964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df82d6c0d06e4b458abc770716d7a62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:23.727118 32202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df82d6c0d06e4b458abc770716d7a62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:23.729121 32693 tablet_bootstrap.cc:492] T df82d6c0d06e4b458abc770716d7a62e P 7ce9b38adfe24c2882b8cfbec30903cb: Bootstrap starting.
I20260809 07:53:23.729121 32692 tablet_bootstrap.cc:492] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042: Bootstrap starting.
I20260809 07:53:23.729542 32409 tablet_service.cc:1467] Processing CreateTablet for tablet df82d6c0d06e4b458abc770716d7a62e (DEFAULT_TABLE table=test-table [id=3ea0d944dee84880af3e763f99971e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:23.729763 32409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df82d6c0d06e4b458abc770716d7a62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:23.729974 32693 tablet_bootstrap.cc:654] T df82d6c0d06e4b458abc770716d7a62e P 7ce9b38adfe24c2882b8cfbec30903cb: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:23.729995 32692 tablet_bootstrap.cc:654] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:23.731060 32693 tablet_bootstrap.cc:492] T df82d6c0d06e4b458abc770716d7a62e P 7ce9b38adfe24c2882b8cfbec30903cb: No bootstrap required, opened a new log
I20260809 07:53:23.731154 32693 ts_tablet_manager.cc:1397] T df82d6c0d06e4b458abc770716d7a62e P 7ce9b38adfe24c2882b8cfbec30903cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:23.731487 32693 raft_consensus.cc:348] T df82d6c0d06e4b458abc770716d7a62e P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:23.731578 32693 raft_consensus.cc:374] T df82d6c0d06e4b458abc770716d7a62e P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:23.731609 32693 raft_consensus.cc:729] T df82d6c0d06e4b458abc770716d7a62e P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce9b38adfe24c2882b8cfbec30903cb, State: Initialized, Role: FOLLOWER
I20260809 07:53:23.731629 32690 tablet_bootstrap.cc:492] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2: Bootstrap starting.
I20260809 07:53:23.731735 32693 consensus_queue.cc:260] T df82d6c0d06e4b458abc770716d7a62e P 7ce9b38adfe24c2882b8cfbec30903cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:23.731974 32693 ts_tablet_manager.cc:1428] T df82d6c0d06e4b458abc770716d7a62e P 7ce9b38adfe24c2882b8cfbec30903cb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 07:53:23.732367 32690 tablet_bootstrap.cc:654] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:23.732620 32692 tablet_bootstrap.cc:492] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042: No bootstrap required, opened a new log
I20260809 07:53:23.732684 32692 ts_tablet_manager.cc:1397] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260809 07:53:23.733069 32692 raft_consensus.cc:348] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:23.733208 32692 raft_consensus.cc:374] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:23.733246 32692 raft_consensus.cc:729] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5f4ab9c0fe4d65b67c36f0cd009042, State: Initialized, Role: FOLLOWER
I20260809 07:53:23.733355 32692 consensus_queue.cc:260] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:23.733451 32690 tablet_bootstrap.cc:492] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2: No bootstrap required, opened a new log
I20260809 07:53:23.733526 32690 ts_tablet_manager.cc:1397] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:23.733604 32692 ts_tablet_manager.cc:1428] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:23.733906 32690 raft_consensus.cc:348] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:23.734048 32690 raft_consensus.cc:374] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:23.734094 32690 raft_consensus.cc:729] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014dc17a9cbe4665a8571a3626c781f2, State: Initialized, Role: FOLLOWER
I20260809 07:53:23.734210 32690 consensus_queue.cc:260] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:23.734459 32690 ts_tablet_manager.cc:1428] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:23.921723 32653 raft_consensus.cc:482] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:23.921859 32653 raft_consensus.cc:504] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:23.922293 32653 leader_election.cc:288] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ce9b38adfe24c2882b8cfbec30903cb (127.28.112.1:36585), 0f5f4ab9c0fe4d65b67c36f0cd009042 (127.28.112.2:34431)
I20260809 07:53:23.922701 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df82d6c0d06e4b458abc770716d7a62e" candidate_uuid: "014dc17a9cbe4665a8571a3626c781f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" is_pre_election: true
I20260809 07:53:23.922708 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df82d6c0d06e4b458abc770716d7a62e" candidate_uuid: "014dc17a9cbe4665a8571a3626c781f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" is_pre_election: true
I20260809 07:53:23.922860 32002 raft_consensus.cc:2454] T df82d6c0d06e4b458abc770716d7a62e P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014dc17a9cbe4665a8571a3626c781f2 in term 0.
I20260809 07:53:23.922863 32228 raft_consensus.cc:2454] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014dc17a9cbe4665a8571a3626c781f2 in term 0.
I20260809 07:53:23.922966 32700 raft_consensus.cc:482] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:23.923077 32700 raft_consensus.cc:504] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:23.923168 32342 leader_election.cc:302] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014dc17a9cbe4665a8571a3626c781f2, 0f5f4ab9c0fe4d65b67c36f0cd009042; no voters:
I20260809 07:53:23.923360 32653 raft_consensus.cc:2781] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:53:23.923413 32653 raft_consensus.cc:482] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:53:23.923456 32653 raft_consensus.cc:3037] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:23.924314 32700 leader_election.cc:288] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ce9b38adfe24c2882b8cfbec30903cb (127.28.112.1:36585), 014dc17a9cbe4665a8571a3626c781f2 (127.28.112.3:36449)
I20260809 07:53:23.924762 32653 raft_consensus.cc:504] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:23.925485 32653 leader_election.cc:288] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [CANDIDATE]: Term 1 election: Requested vote from peers 7ce9b38adfe24c2882b8cfbec30903cb (127.28.112.1:36585), 0f5f4ab9c0fe4d65b67c36f0cd009042 (127.28.112.2:34431)
I20260809 07:53:23.925650 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df82d6c0d06e4b458abc770716d7a62e" candidate_uuid: "014dc17a9cbe4665a8571a3626c781f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce9b38adfe24c2882b8cfbec30903cb"
I20260809 07:53:23.925778 32002 raft_consensus.cc:3037] T df82d6c0d06e4b458abc770716d7a62e P 7ce9b38adfe24c2882b8cfbec30903cb [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:23.926704 32228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df82d6c0d06e4b458abc770716d7a62e" candidate_uuid: "014dc17a9cbe4665a8571a3626c781f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042"
I20260809 07:53:23.926819 32228 raft_consensus.cc:3037] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:23.926923 32002 raft_consensus.cc:2454] T df82d6c0d06e4b458abc770716d7a62e P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014dc17a9cbe4665a8571a3626c781f2 in term 1.
I20260809 07:53:23.927227 32340 leader_election.cc:302] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014dc17a9cbe4665a8571a3626c781f2, 7ce9b38adfe24c2882b8cfbec30903cb; no voters:
I20260809 07:53:23.927385 32653 raft_consensus.cc:2781] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:23.927685 32653 raft_consensus.cc:686] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [term 1 LEADER]: Becoming Leader. State: Replica: 014dc17a9cbe4665a8571a3626c781f2, State: Running, Role: LEADER
I20260809 07:53:23.927839 32228 raft_consensus.cc:2454] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014dc17a9cbe4665a8571a3626c781f2 in term 1.
I20260809 07:53:23.927837 32653 consensus_queue.cc:237] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } }
I20260809 07:53:23.928910 32715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:23.924478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:36449 (local address 127.28.112.2:47389)
0809 07:53:23.924782 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:23.924798 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:23.924906 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:23.925461 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:23.925464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:23.925483 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:23.925603 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:23.925610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:23.926512 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:23.926516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:23.927585 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:23.927596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:23.928431 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:23.928435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:23.928593 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:23.928621 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 07:53:23.928683 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 07:53:23.928754 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":34,"thread_start_us":67,"threads_started":1}
I20260809 07:53:23.928912 32723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:23.924976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:47389 (local address 127.28.112.3:36449)
0809 07:53:23.925213 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:23.925227 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:23.925241 (+ 14us) server_negotiation.cc:407] Connection header received
0809 07:53:23.925289 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:23.925291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:23.925334 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:23.925699 (+ 365us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:23.925791 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:23.926399 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:23.927722 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:23.928292 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:23.928624 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:23.928679 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:23.928714 (+ 35us) server_negotiation.cc:299] Negotiation successful
0809 07:53:23.928779 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":84,"threads_started":1}
I20260809 07:53:23.929569 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:23.924739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:42063 (local address 127.28.112.1:36585)
0809 07:53:23.924863 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:23.924867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:23.924985 (+ 118us) server_negotiation.cc:407] Connection header received
0809 07:53:23.925100 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:23.925103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:23.925148 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:23.925208 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:23.925486 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:23.926214 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:23.927653 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:23.928267 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:23.928356 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:23.928414 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:23.929354 (+ 940us) server_negotiation.cc:299] Negotiation successful
0809 07:53:23.929467 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 07:53:23.929594 32716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:23.924478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:36585 (local address 127.28.112.2:42063)
0809 07:53:23.924905 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:23.924918 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:23.925017 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:23.925224 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:23.925227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:23.925242 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:23.925362 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:23.925369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:23.926315 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:23.926318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:23.927510 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:23.927521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:23.928512 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:23.928515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:23.929206 (+ 691us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:23.929232 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 07:53:23.929318 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 07:53:23.929388 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":43,"thread_start_us":88,"threads_started":1}
I20260809 07:53:23.929929 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df82d6c0d06e4b458abc770716d7a62e" candidate_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014dc17a9cbe4665a8571a3626c781f2" is_pre_election: true
I20260809 07:53:23.930025 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df82d6c0d06e4b458abc770716d7a62e" candidate_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" is_pre_election: true
I20260809 07:53:23.930174 32002 raft_consensus.cc:2379] T df82d6c0d06e4b458abc770716d7a62e P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0f5f4ab9c0fe4d65b67c36f0cd009042 in current term 1: Already voted for candidate 014dc17a9cbe4665a8571a3626c781f2 in this term.
I20260809 07:53:23.930047 31750 catalog_manager.cc:5495] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 014dc17a9cbe4665a8571a3626c781f2 (127.28.112.3). New cstate: current_term: 1 leader_uuid: "014dc17a9cbe4665a8571a3626c781f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "014dc17a9cbe4665a8571a3626c781f2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36449 } health_report { overall_health: HEALTHY } } }
I20260809 07:53:23.930538 32116 leader_election.cc:302] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f5f4ab9c0fe4d65b67c36f0cd009042; no voters: 014dc17a9cbe4665a8571a3626c781f2, 7ce9b38adfe24c2882b8cfbec30903cb
I20260809 07:53:23.930794 32700 raft_consensus.cc:2726] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 07:53:23.979239 32723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:23.971053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.28.112.3:36449)
0809 07:53:23.971363 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:23.971368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:23.971379 (+ 11us) server_negotiation.cc:407] Connection header received
0809 07:53:23.971420 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:23.971422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:23.971468 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:23.971532 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:23.971765 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:23.974259 (+ 2494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:23.975324 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:23.975672 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:23.975768 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:23.976453 (+ 685us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:23.977187 (+ 734us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:23.977747 (+ 560us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:23.978949 (+ 1202us) server_negotiation.cc:299] Negotiation successful
0809 07:53:23.979111 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239}
I20260809 07:53:23.992170 32723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:23.987258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:51979 (local address 127.28.112.3:36449)
0809 07:53:23.987360 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:23.987364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:23.987484 (+ 120us) server_negotiation.cc:407] Connection header received
0809 07:53:23.987586 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:23.987588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:23.987626 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:23.987682 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:23.987918 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:23.988400 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:23.989530 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:23.990077 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:23.990364 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:23.991561 (+ 1197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:23.991696 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:23.991814 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:23.992004 (+ 190us) server_negotiation.cc:299] Negotiation successful
0809 07:53:23.992068 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":40}
I20260809 07:53:23.992182 32729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:23.987174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:36449 (local address 127.28.112.3:51979)
0809 07:53:23.987407 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:23.987420 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:23.987514 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:23.987701 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:23.987703 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:23.987710 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 07:53:23.987801 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:23.987805 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:23.988494 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:23.988497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:23.989411 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:23.989421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:23.990516 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:23.990574 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:23.991171 (+ 597us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:23.991252 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 07:53:23.991847 (+ 595us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 07:53:23.991859 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 07:53:23.991978 (+ 119us) client_negotiation.cc:240] Negotiation successful
0809 07:53:23.992060 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260809 07:53:23.997221 32228 raft_consensus.cc:1261] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 1 FOLLOWER]: Refusing update from remote peer 014dc17a9cbe4665a8571a3626c781f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:23.997318 32002 raft_consensus.cc:1261] T df82d6c0d06e4b458abc770716d7a62e P 7ce9b38adfe24c2882b8cfbec30903cb [term 1 FOLLOWER]: Refusing update from remote peer 014dc17a9cbe4665a8571a3626c781f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:23.997589 32725 consensus_queue.cc:1035] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f5f4ab9c0fe4d65b67c36f0cd009042" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:23.997776 32643 consensus_queue.cc:1035] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ce9b38adfe24c2882b8cfbec30903cb" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 07:53:24.010324 32641 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 07:53:25.559911 31752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:25.560053 31752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:25.562309 31752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a01ba57d6e004e4991c7a10dd11dce14: Sending DeleteTablet for 3 replicas of tablet df82d6c0d06e4b458abc770716d7a62e
I20260809 07:53:25.562896 32202 tablet_service.cc:1514] Processing DeleteTablet for tablet df82d6c0d06e4b458abc770716d7a62e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:25 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:57090
I20260809 07:53:25.562896 31964 tablet_service.cc:1514] Processing DeleteTablet for tablet df82d6c0d06e4b458abc770716d7a62e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:25 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:53368
I20260809 07:53:25.562994 32409 tablet_service.cc:1514] Processing DeleteTablet for tablet df82d6c0d06e4b458abc770716d7a62e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:25 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:35100
I20260809 07:53:25.563333 32762 tablet_replica.cc:330] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2: stopping tablet replica
I20260809 07:53:25.563354 32760 tablet_replica.cc:330] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042: stopping tablet replica
I20260809 07:53:25.563460 32762 raft_consensus.cc:2229] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:53:25.563465 32760 raft_consensus.cc:2229] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:25.563522 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 31869
I20260809 07:53:25.563591 32760 raft_consensus.cc:2258] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:25.568708 32762 raft_consensus.cc:2258] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:25.569092 32760 ts_tablet_manager.cc:1905] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:25.569232 32762 ts_tablet_manager.cc:1905] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:25.571393 32762 ts_tablet_manager.cc:1918] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 07:53:25.571465 32762 log.cc:1196] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/df82d6c0d06e4b458abc770716d7a62e
I20260809 07:53:25.571759 32762 ts_tablet_manager.cc:1939] T df82d6c0d06e4b458abc770716d7a62e P 014dc17a9cbe4665a8571a3626c781f2: Deleting consensus metadata
I20260809 07:53:25.571781 32760 ts_tablet_manager.cc:1918] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 07:53:25.571875 32760 log.cc:1196] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/df82d6c0d06e4b458abc770716d7a62e
I20260809 07:53:25.572160 32760 ts_tablet_manager.cc:1939] T df82d6c0d06e4b458abc770716d7a62e P 0f5f4ab9c0fe4d65b67c36f0cd009042: Deleting consensus metadata
I20260809 07:53:25.572372 31722 catalog_manager.cc:4841] TS 014dc17a9cbe4665a8571a3626c781f2 (127.28.112.3:36449): tablet df82d6c0d06e4b458abc770716d7a62e (table test-table [id=3ea0d944dee84880af3e763f99971e37]) successfully deleted
I20260809 07:53:25.572880 31724 catalog_manager.cc:4841] TS 0f5f4ab9c0fe4d65b67c36f0cd009042 (127.28.112.2:34431): tablet df82d6c0d06e4b458abc770716d7a62e (table test-table [id=3ea0d944dee84880af3e763f99971e37]) successfully deleted
I20260809 07:53:25.579540 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 32086
W20260809 07:53:25.579545 31723 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.112.1:36585 (error 108)
I20260809 07:53:25.584100 32765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:25.582968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:36585 (local address 127.0.0.1:53398)
0809 07:53:25.583938 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:25.584011 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":896,"thread_start_us":482,"threads_started":1}
W20260809 07:53:25.584245 31723 catalog_manager.cc:4584] TS 7ce9b38adfe24c2882b8cfbec30903cb (127.28.112.1:36585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet df82d6c0d06e4b458abc770716d7a62e: Network error: Client connection negotiation failed: client connection to 127.28.112.1:36585: connect: Connection refused (error 111)
I20260809 07:53:25.588385 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 32316
I20260809 07:53:25.597656 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 31700
2026-08-09T07:53:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6792 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:53:25 dist-test-slave-vpvm krb5kdc[302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:53:25 dist-test-slave-vpvm krb5kdc[302](info): set up 2 sockets
Aug 09 07:53:25 dist-test-slave-vpvm krb5kdc[302](info): commencing operation
krb5kdc: starting...
W20260809 07:53:26.731770 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:53:26 dist-test-slave-vpvm krb5kdc[302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:53:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:53:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:53:26.881418 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:42417
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:46683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:42417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:26.990553 320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:26.990806 320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:26.990845 320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:26.994709 320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:26.994763 320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:26.994781 320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:26.994802 320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:26.994817 320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:53:27.000041 320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:46683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:42417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:42417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:27.001047 320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:27.002063 320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:27.010401 333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:27.010460 320 server_base.cc:1034] running on GCE node
W20260809 07:53:27.010622 328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:27.010509 331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:27.011251 320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:27.012338 320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:27.013495 320 hybrid_clock.cc:648] HybridClock initialized: now 1786262007013482 us; error 30 us; skew 500 ppm
Aug 09 07:53:27 dist-test-slave-vpvm krb5kdc[302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262007, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:27.016120 320 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:53:27.016924 320 webserver.cc:460] Webserver started at http://127.28.112.62:37223/ using document root <none> and password file <none>
I20260809 07:53:27.017418 320 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:27.017475 320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:27.017643 320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:27.019577 320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "06ef93f7c2de48fabb980ddacf41ae43"
format_stamp: "Formatted at 2026-08-09 07:53:27 on dist-test-slave-vpvm"
server_key: "4a1a15ed4d606a936f92da3e8f1d2c87"
server_key_iv: "cf0030fe52dc2f546e10b359f3e71b2e"
server_key_version: "encryptionkey@0"
I20260809 07:53:27.020093 320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "06ef93f7c2de48fabb980ddacf41ae43"
format_stamp: "Formatted at 2026-08-09 07:53:27 on dist-test-slave-vpvm"
server_key: "4a1a15ed4d606a936f92da3e8f1d2c87"
server_key_iv: "cf0030fe52dc2f546e10b359f3e71b2e"
server_key_version: "encryptionkey@0"
I20260809 07:53:27.023569 320 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260809 07:53:27.025981 344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:27.027184 320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 07:53:27.027292 320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "06ef93f7c2de48fabb980ddacf41ae43"
format_stamp: "Formatted at 2026-08-09 07:53:27 on dist-test-slave-vpvm"
server_key: "4a1a15ed4d606a936f92da3e8f1d2c87"
server_key_iv: "cf0030fe52dc2f546e10b359f3e71b2e"
server_key_version: "encryptionkey@0"
I20260809 07:53:27.027376 320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:27.045913 320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:27.048686 320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:27.048838 320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:27.055745 320 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:42417
I20260809 07:53:27.055771 414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:42417 every 8 connection(s)
I20260809 07:53:27.056746 320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:53:27.058573 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 320
I20260809 07:53:27.058696 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:53:27.059039 29120 external_mini_cluster.cc:1428] Setting key 60303fc7674a40b945b8f014a53706ad
I20260809 07:53:27.061208 415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:27.068061 415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43: Bootstrap starting.
Aug 09 07:53:27 dist-test-slave-vpvm krb5kdc[302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:27.070786 415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:27.071636 415 log.cc:824] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:27.073882 415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43: No bootstrap required, opened a new log
I20260809 07:53:27.073985 422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:27.060467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52108 (local address 127.28.112.62:42417)
0809 07:53:27.060922 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:27.060944 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:27.060988 (+ 44us) server_negotiation.cc:407] Connection header received
0809 07:53:27.061853 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:27.061875 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:27.062330 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:27.062699 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:27.063381 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:27.064866 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:27.065682 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:27.066390 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:27.066568 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:27.068925 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:27.068932 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:27.068947 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:27.068983 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:27.070903 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:27.071324 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:27.071330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:27.071336 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:27.071493 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:27.071738 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:27.071743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:27.071745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:27.072221 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:27.072371 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:27.072699 (+ 328us) server_negotiation.cc:299] Negotiation successful
0809 07:53:27.073016 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":139,"threads_started":1}
I20260809 07:53:27.077445 415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ef93f7c2de48fabb980ddacf41ae43" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 42417 } }
I20260809 07:53:27.077658 415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:27.077703 415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ef93f7c2de48fabb980ddacf41ae43, State: Initialized, Role: FOLLOWER
I20260809 07:53:27.078099 415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ef93f7c2de48fabb980ddacf41ae43" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 42417 } }
I20260809 07:53:27.078229 415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:53:27.078279 415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:53:27.078353 415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:27.079171 415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ef93f7c2de48fabb980ddacf41ae43" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 42417 } }
I20260809 07:53:27.079476 415 leader_election.cc:302] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06ef93f7c2de48fabb980ddacf41ae43; no voters:
I20260809 07:53:27.079743 415 leader_election.cc:288] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:53:27.079855 425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:27.080067 425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43 [term 1 LEADER]: Becoming Leader. State: Replica: 06ef93f7c2de48fabb980ddacf41ae43, State: Running, Role: LEADER
I20260809 07:53:27.080415 425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ef93f7c2de48fabb980ddacf41ae43" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 42417 } }
I20260809 07:53:27.080864 415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:27.082093 427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06ef93f7c2de48fabb980ddacf41ae43. Latest consensus state: current_term: 1 leader_uuid: "06ef93f7c2de48fabb980ddacf41ae43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ef93f7c2de48fabb980ddacf41ae43" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 42417 } } }
I20260809 07:53:27.082217 427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:27.082495 426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06ef93f7c2de48fabb980ddacf41ae43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ef93f7c2de48fabb980ddacf41ae43" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 42417 } } }
I20260809 07:53:27.082612 426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:27.082667 437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:53:27.085930 437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:27.091801 437 catalog_manager.cc:1347] Generated new cluster ID: afc1576cc6df4c90bedbd64f389f564a
I20260809 07:53:27.091912 437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:27.111276 437 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:53:27.112241 437 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:27.119092 437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06ef93f7c2de48fabb980ddacf41ae43: Generated new TSK 0
I20260809 07:53:27.119803 437 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
I20260809 07:53:27.156466 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:42417
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:46683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:27.275490 451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:27.275745 451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:27.275789 451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:27.279649 451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:27.279691 451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:27.279744 451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:53:27.284827 451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:46683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:42417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:27.285750 451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:27.286725 451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:27.295961 458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:27.296002 456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:27.295953 462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:27.296164 451 server_base.cc:1034] running on GCE node
I20260809 07:53:27.296696 451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:27.297266 451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:27.298413 451 hybrid_clock.cc:648] HybridClock initialized: now 1786262007298396 us; error 29 us; skew 500 ppm
Aug 09 07:53:27 dist-test-slave-vpvm krb5kdc[302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262007, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:27.301149 451 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:53:27.302054 451 webserver.cc:460] Webserver started at http://127.28.112.1:32879/ using document root <none> and password file <none>
I20260809 07:53:27.302588 451 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:27.302655 451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:27.302850 451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:27.304827 451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "e51e59ba7b9b4b07b10063093482db7b"
format_stamp: "Formatted at 2026-08-09 07:53:27 on dist-test-slave-vpvm"
server_key: "0b69cb63212e3ba0cf42020fd4d3cbaf"
server_key_iv: "404ffb87d292162dbd2daffb355e1483"
server_key_version: "encryptionkey@0"
I20260809 07:53:27.305347 451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "e51e59ba7b9b4b07b10063093482db7b"
format_stamp: "Formatted at 2026-08-09 07:53:27 on dist-test-slave-vpvm"
server_key: "0b69cb63212e3ba0cf42020fd4d3cbaf"
server_key_iv: "404ffb87d292162dbd2daffb355e1483"
server_key_version: "encryptionkey@0"
I20260809 07:53:27.308833 451 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:53:27.311071 470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:27.312235 451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:27.312381 451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "e51e59ba7b9b4b07b10063093482db7b"
format_stamp: "Formatted at 2026-08-09 07:53:27 on dist-test-slave-vpvm"
server_key: "0b69cb63212e3ba0cf42020fd4d3cbaf"
server_key_iv: "404ffb87d292162dbd2daffb355e1483"
server_key_version: "encryptionkey@0"
I20260809 07:53:27.312501 451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:27.339627 451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:27.342307 451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:27.342491 451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:27.343070 451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:27.343897 451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:27.343945 451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:27.343998 451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:27.344023 451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:27.352286 451 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:35593
I20260809 07:53:27.352315 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:35593 every 8 connection(s)
I20260809 07:53:27.353173 451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:53:27.353451 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 451
I20260809 07:53:27.353549 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:53:27.353881 29120 external_mini_cluster.cc:1428] Setting key 2143e1490b04118ae5682825fef9e185
Aug 09 07:53:27 dist-test-slave-vpvm krb5kdc[302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262007, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.112.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.2@KRBTEST.COM" created.
I20260809 07:53:27.366460 422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:27.354988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:35465 (local address 127.28.112.62:42417)
0809 07:53:27.355131 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:27.355136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:27.356061 (+ 925us) server_negotiation.cc:407] Connection header received
0809 07:53:27.357379 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:27.357383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:27.357437 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:27.357523 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:27.358674 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:27.359200 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:27.360333 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:27.360573 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:27.360661 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:27.363245 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:27.363248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:27.363250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:27.363274 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:27.364509 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:27.364952 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:27.364955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:27.364956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:27.365018 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:27.365335 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:27.365340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:27.365341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:27.365588 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:27.365680 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:27.366165 (+ 485us) server_negotiation.cc:299] Negotiation successful
0809 07:53:27.366303 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 07:53:27.367254 638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:27.355400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:42417 (local address 127.28.112.1:35465)
0809 07:53:27.355884 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:27.355945 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:27.357178 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:27.357663 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:27.357669 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:27.358070 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:27.358533 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:27.358546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:27.359315 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:27.359319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:27.360220 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:27.360231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:27.360708 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:27.360712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:27.360865 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:27.361382 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:27.361398 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:27.363056 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:27.364620 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:27.364625 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:27.364637 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:27.364878 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:27.365076 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:27.365078 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:27.365080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:27.365207 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:27.365676 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:27.365680 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:27.365890 (+ 210us) client_negotiation.cc:769] Sending connection context
0809 07:53:27.366097 (+ 207us) client_negotiation.cc:240] Negotiation successful
0809 07:53:27.366437 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":123,"threads_started":1}
I20260809 07:53:27.368683 635 heartbeater.cc:347] Connected to a master server at 127.28.112.62:42417
I20260809 07:53:27.368981 635 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:27.369544 635 heartbeater.cc:511] Master 127.28.112.62:42417 requested a full tablet report, sending...
I20260809 07:53:27.371268 368 ts_manager.cc:194] Registered new tserver with Master: e51e59ba7b9b4b07b10063093482db7b (127.28.112.1:35593)
I20260809 07:53:27.372407 368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:35465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
I20260809 07:53:27.380863 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:42417
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:46683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:27.489064 643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:27.489300 643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:27.489356 643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:27.493235 643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:27.493283 643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:27.493360 643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:53:27.498272 643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:46683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:42417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:27.499222 643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:27.500168 643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:27.509115 653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:27.509128 654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:27.509181 656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:27.509410 643 server_base.cc:1034] running on GCE node
I20260809 07:53:27.509807 643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:27.510434 643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:27.511600 643 hybrid_clock.cc:648] HybridClock initialized: now 1786262007511585 us; error 32 us; skew 500 ppm
Aug 09 07:53:27 dist-test-slave-vpvm krb5kdc[302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262007, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:27.514200 643 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:53:27.515026 643 webserver.cc:460] Webserver started at http://127.28.112.2:42123/ using document root <none> and password file <none>
I20260809 07:53:27.515544 643 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:27.515614 643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:27.515774 643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:27.517716 643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "b32c69f2755c439bbee51171caf24ae4"
format_stamp: "Formatted at 2026-08-09 07:53:27 on dist-test-slave-vpvm"
server_key: "066c19145a3edc1d1c4d2b67c101ce27"
server_key_iv: "1e1dad6d88144dd6f8b8740fc6b7aa5e"
server_key_version: "encryptionkey@0"
I20260809 07:53:27.518263 643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "b32c69f2755c439bbee51171caf24ae4"
format_stamp: "Formatted at 2026-08-09 07:53:27 on dist-test-slave-vpvm"
server_key: "066c19145a3edc1d1c4d2b67c101ce27"
server_key_iv: "1e1dad6d88144dd6f8b8740fc6b7aa5e"
server_key_version: "encryptionkey@0"
I20260809 07:53:27.521751 643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 07:53:27.524155 666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:27.525301 643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:27.525411 643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "b32c69f2755c439bbee51171caf24ae4"
format_stamp: "Formatted at 2026-08-09 07:53:27 on dist-test-slave-vpvm"
server_key: "066c19145a3edc1d1c4d2b67c101ce27"
server_key_iv: "1e1dad6d88144dd6f8b8740fc6b7aa5e"
server_key_version: "encryptionkey@0"
I20260809 07:53:27.525511 643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:27.536613 643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:27.539510 643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:27.539687 643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:27.540223 643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:27.541018 643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:27.541064 643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:27.541105 643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:27.541134 643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:27.549486 643 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:37817
I20260809 07:53:27.549520 824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:37817 every 8 connection(s)
I20260809 07:53:27.550323 643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:53:27.557260 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 643
I20260809 07:53:27.557379 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:53:27.557689 29120 external_mini_cluster.cc:1428] Setting key 2c46333e7014f6373667014deb2be40d
Aug 09 07:53:27 dist-test-slave-vpvm krb5kdc[302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262007, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:27.564401 422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:27.552016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:59667 (local address 127.28.112.62:42417)
0809 07:53:27.552155 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:27.552159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:27.553002 (+ 843us) server_negotiation.cc:407] Connection header received
0809 07:53:27.554297 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:27.554300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:27.554348 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:27.554424 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:27.555503 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:27.556029 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:27.557100 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:27.557436 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:27.557544 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:27.561033 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:27.561036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:27.561038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:27.561063 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:27.562383 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:27.562910 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:27.562912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:27.562913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:27.562986 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:27.563330 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:27.563332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:27.563333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:27.563492 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:27.563563 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:27.564166 (+ 603us) server_negotiation.cc:299] Negotiation successful
0809 07:53:27.564283 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 07:53:27.565531 831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:27.552404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:42417 (local address 127.28.112.2:59667)
0809 07:53:27.552851 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:27.552894 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:27.554091 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:27.554562 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:27.554568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:27.554900 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:27.555353 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:27.555366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:27.556126 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:27.556128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:27.556976 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:27.556985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:27.557649 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:27.557655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:27.558083 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:27.559144 (+ 1061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:27.559164 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:27.560866 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:27.562494 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:27.562500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:27.562515 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:27.562822 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:27.563067 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:27.563070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:27.563072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:27.563251 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:27.563581 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:27.563588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:27.563901 (+ 313us) client_negotiation.cc:769] Sending connection context
0809 07:53:27.564131 (+ 230us) client_negotiation.cc:240] Negotiation successful
0809 07:53:27.564469 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":104,"threads_started":1}
I20260809 07:53:27.566875 825 heartbeater.cc:347] Connected to a master server at 127.28.112.62:42417
I20260809 07:53:27.567102 825 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:27.567579 825 heartbeater.cc:511] Master 127.28.112.62:42417 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.112.3@KRBTEST.COM; defaulting to no policy
I20260809 07:53:27.568666 368 ts_manager.cc:194] Registered new tserver with Master: b32c69f2755c439bbee51171caf24ae4 (127.28.112.2:37817)
I20260809 07:53:27.569216 368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:59667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
I20260809 07:53:27.584486 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:42417
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:46683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:27.694752 836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:27.694994 836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:27.695050 836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:27.698948 836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:27.699004 836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:27.699088 836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:53:27.704066 836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:46683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:42417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:27.704986 836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:27.705976 836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:27.714870 853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:27.714891 847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:27.714960 849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:27.715011 836 server_base.cc:1034] running on GCE node
I20260809 07:53:27.715480 836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:27.716120 836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:27.717290 836 hybrid_clock.cc:648] HybridClock initialized: now 1786262007717262 us; error 49 us; skew 500 ppm
Aug 09 07:53:27 dist-test-slave-vpvm krb5kdc[302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262007, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:27.719942 836 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:53:27.720731 836 webserver.cc:460] Webserver started at http://127.28.112.3:43713/ using document root <none> and password file <none>
I20260809 07:53:27.721240 836 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:27.721306 836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:27.721498 836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:27.723440 836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "a2ff8972d6454eba8115689f08a89003"
format_stamp: "Formatted at 2026-08-09 07:53:27 on dist-test-slave-vpvm"
server_key: "32103591146edb9cc448fd79744d0e29"
server_key_iv: "5be632d95a51fadc1a6f993a4627607a"
server_key_version: "encryptionkey@0"
I20260809 07:53:27.723949 836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "a2ff8972d6454eba8115689f08a89003"
format_stamp: "Formatted at 2026-08-09 07:53:27 on dist-test-slave-vpvm"
server_key: "32103591146edb9cc448fd79744d0e29"
server_key_iv: "5be632d95a51fadc1a6f993a4627607a"
server_key_version: "encryptionkey@0"
I20260809 07:53:27.727510 836 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260809 07:53:27.729734 863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:27.730863 836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 07:53:27.730975 836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "a2ff8972d6454eba8115689f08a89003"
format_stamp: "Formatted at 2026-08-09 07:53:27 on dist-test-slave-vpvm"
server_key: "32103591146edb9cc448fd79744d0e29"
server_key_iv: "5be632d95a51fadc1a6f993a4627607a"
server_key_version: "encryptionkey@0"
I20260809 07:53:27.731055 836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:27.749738 836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:27.752251 836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:27.752414 836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:27.752954 836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:27.753721 836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:27.753768 836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:27.753820 836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:27.753849 836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:27.762148 836 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:43585
I20260809 07:53:27.762207 1033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:43585 every 8 connection(s)
I20260809 07:53:27.763041 836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:53:27.769923 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 836
I20260809 07:53:27.770061 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:53:27.770370 29120 external_mini_cluster.cc:1428] Setting key 183a1fbb3e44f1b6ee62d7535e672403
Aug 09 07:53:27 dist-test-slave-vpvm krb5kdc[302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262007, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:27.775250 422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:27.764756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:50015 (local address 127.28.112.62:42417)
0809 07:53:27.764894 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:27.764897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:27.765696 (+ 799us) server_negotiation.cc:407] Connection header received
0809 07:53:27.766954 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:27.766957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:27.767003 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:27.767075 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:27.768126 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:27.768635 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:27.769684 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:27.769878 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:27.769971 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:27.772490 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:27.772494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:27.772496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:27.772523 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:27.773599 (+ 1076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:27.774059 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:27.774061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:27.774063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:27.774134 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:27.774403 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:27.774405 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:27.774406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:27.774576 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:27.774646 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:27.775044 (+ 398us) server_negotiation.cc:299] Negotiation successful
0809 07:53:27.775136 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 07:53:27.775978 1038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:27.765098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:42417 (local address 127.28.112.3:50015)
0809 07:53:27.765548 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:27.765594 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:27.766771 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:27.767214 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:27.767220 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:27.767558 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:27.768017 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:27.768030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:27.768741 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:27.768743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:27.769598 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:27.769607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:27.770062 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:27.770066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:27.770253 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:27.770773 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:27.770789 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:27.772324 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:27.773703 (+ 1379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:27.773708 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:27.773719 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:27.773989 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:27.774198 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:27.774200 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:27.774202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:27.774333 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:27.774653 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:27.774658 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:27.774863 (+ 205us) client_negotiation.cc:769] Sending connection context
0809 07:53:27.775002 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 07:53:27.775232 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20260809 07:53:27.776970 1034 heartbeater.cc:347] Connected to a master server at 127.28.112.62:42417
I20260809 07:53:27.777189 1034 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:27.777630 1034 heartbeater.cc:511] Master 127.28.112.62:42417 requested a full tablet report, sending...
I20260809 07:53:27.778622 368 ts_manager.cc:194] Registered new tserver with Master: a2ff8972d6454eba8115689f08a89003 (127.28.112.3:43585)
I20260809 07:53:27.779151 368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:50015
I20260809 07:53:27.783923 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 07:53:27.796551 422 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:27.795364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52114 (local address 127.28.112.62:42417)
0809 07:53:27.795503 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:27.795508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:27.795640 (+ 132us) server_negotiation.cc:407] Connection header received
0809 07:53:27.795739 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:27.795742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:27.795779 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:27.795842 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:27.796431 (+ 589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52114: BlockingRecv error: recv got EOF from 127.0.0.1:52114 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 07:53:27.797596 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 451
I20260809 07:53:27.821566 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 643
I20260809 07:53:27.827698 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 836
I20260809 07:53:27.836560 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 320
2026-08-09T07:53:27Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2236 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:53:27 dist-test-slave-vpvm krb5kdc[1054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:53:27 dist-test-slave-vpvm krb5kdc[1054](info): set up 2 sockets
Aug 09 07:53:27 dist-test-slave-vpvm krb5kdc[1054](info): commencing operation
krb5kdc: starting...
W20260809 07:53:28.970113 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:53:29 dist-test-slave-vpvm krb5kdc[1054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:53:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:53:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:53:29.085393 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:46743
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:46743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:29.191885 1070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:29.192107 1070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:29.192160 1070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:29.196044 1070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:29.196094 1070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:29.196110 1070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:29.196125 1070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:29.196161 1070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:53:29.201129 1070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:46743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:46743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:29.202080 1070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:29.203013 1070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:29.211187 1082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:29.211187 1080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:29.211329 1070 server_base.cc:1034] running on GCE node
W20260809 07:53:29.211166 1078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:29.211978 1070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:29.213017 1070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:29.214164 1070 hybrid_clock.cc:648] HybridClock initialized: now 1786262009214144 us; error 37 us; skew 500 ppm
Aug 09 07:53:29 dist-test-slave-vpvm krb5kdc[1054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262009, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:29.220245 1070 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:53:29.221095 1070 webserver.cc:460] Webserver started at http://127.28.112.62:37499/ using document root <none> and password file <none>
I20260809 07:53:29.221596 1070 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:29.221670 1070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:29.221870 1070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:29.223840 1070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "d76c5f1b29194efba267183c5caa7c6e"
format_stamp: "Formatted at 2026-08-09 07:53:29 on dist-test-slave-vpvm"
server_key: "222e9fbd632bca6c3285448099e1b57a"
server_key_iv: "12c4d5326e11951622dc6790bf22a568"
server_key_version: "encryptionkey@0"
I20260809 07:53:29.224362 1070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "d76c5f1b29194efba267183c5caa7c6e"
format_stamp: "Formatted at 2026-08-09 07:53:29 on dist-test-slave-vpvm"
server_key: "222e9fbd632bca6c3285448099e1b57a"
server_key_iv: "12c4d5326e11951622dc6790bf22a568"
server_key_version: "encryptionkey@0"
I20260809 07:53:29.227867 1070 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 07:53:29.230078 1093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:29.231233 1070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:29.231348 1070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "d76c5f1b29194efba267183c5caa7c6e"
format_stamp: "Formatted at 2026-08-09 07:53:29 on dist-test-slave-vpvm"
server_key: "222e9fbd632bca6c3285448099e1b57a"
server_key_iv: "12c4d5326e11951622dc6790bf22a568"
server_key_version: "encryptionkey@0"
I20260809 07:53:29.231434 1070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:29.253705 1070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:29.256865 1070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:29.257057 1070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:29.264268 1070 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:46743
I20260809 07:53:29.264535 1172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:46743 every 8 connection(s)
I20260809 07:53:29.265297 1070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:53:29.268023 1173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:29.271047 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 1070
I20260809 07:53:29.271152 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:53:29.271399 29120 external_mini_cluster.cc:1428] Setting key 0804b5974901e04618af6eaab3cb9f50
I20260809 07:53:29.273491 1173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e: Bootstrap starting.
I20260809 07:53:29.275942 1173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:29.276716 1173 log.cc:824] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:29.278638 1173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e: No bootstrap required, opened a new log
Aug 09 07:53:29 dist-test-slave-vpvm krb5kdc[1054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:29.282280 1173 raft_consensus.cc:348] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76c5f1b29194efba267183c5caa7c6e" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 46743 } }
I20260809 07:53:29.282490 1173 raft_consensus.cc:374] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:29.282549 1173 raft_consensus.cc:729] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d76c5f1b29194efba267183c5caa7c6e, State: Initialized, Role: FOLLOWER
I20260809 07:53:29.282963 1173 consensus_queue.cc:260] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76c5f1b29194efba267183c5caa7c6e" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 46743 } }
I20260809 07:53:29.283083 1173 raft_consensus.cc:388] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:53:29.283161 1173 raft_consensus.cc:482] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:53:29.283260 1173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:29.284078 1173 raft_consensus.cc:504] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76c5f1b29194efba267183c5caa7c6e" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 46743 } }
I20260809 07:53:29.284401 1173 leader_election.cc:302] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d76c5f1b29194efba267183c5caa7c6e; no voters:
I20260809 07:53:29.284665 1173 leader_election.cc:288] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:53:29.284759 1178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:29.284958 1176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:29.272649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34312 (local address 127.28.112.62:46743)
0809 07:53:29.273025 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:29.273040 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:29.273076 (+ 36us) server_negotiation.cc:407] Connection header received
0809 07:53:29.273687 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:29.273705 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:29.274051 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:29.274392 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:29.275242 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:29.276269 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:29.277351 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:29.277800 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:29.277944 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:29.280503 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:29.280509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:29.280521 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:29.280546 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:29.282068 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:29.282458 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:29.282463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:29.282469 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:29.282629 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:29.282892 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:29.282895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:29.282897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:29.283356 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:29.283563 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:29.283827 (+ 264us) server_negotiation.cc:299] Negotiation successful
0809 07:53:29.284101 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":95,"threads_started":1}
I20260809 07:53:29.284963 1178 raft_consensus.cc:686] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e [term 1 LEADER]: Becoming Leader. State: Replica: d76c5f1b29194efba267183c5caa7c6e, State: Running, Role: LEADER
I20260809 07:53:29.285524 1178 consensus_queue.cc:237] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76c5f1b29194efba267183c5caa7c6e" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 46743 } }
I20260809 07:53:29.285832 1173 sys_catalog.cc:564] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:29.287873 1181 sys_catalog.cc:455] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d76c5f1b29194efba267183c5caa7c6e. Latest consensus state: current_term: 1 leader_uuid: "d76c5f1b29194efba267183c5caa7c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76c5f1b29194efba267183c5caa7c6e" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 46743 } } }
I20260809 07:53:29.288007 1181 sys_catalog.cc:458] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:29.288491 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:53:29.288537 1180 sys_catalog.cc:455] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d76c5f1b29194efba267183c5caa7c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76c5f1b29194efba267183c5caa7c6e" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 46743 } } }
I20260809 07:53:29.288617 1180 sys_catalog.cc:458] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e [sys.catalog]: This master's current role is: LEADER
W20260809 07:53:29.290808 1201 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:53:29.290872 1201 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:53:29.290928 1203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:53:29.291603 1203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:29.296772 1203 catalog_manager.cc:1347] Generated new cluster ID: 9cc9a72d34f6423b8bfbb672cae5fcad
I20260809 07:53:29.296824 1203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:29.306728 1203 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:53:29.307796 1203 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:29.309270 1176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:29.303848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34324 (local address 127.28.112.62:46743)
0809 07:53:29.303962 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:29.303966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:29.304234 (+ 268us) server_negotiation.cc:407] Connection header received
0809 07:53:29.304379 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:29.304382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:29.304443 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:29.304520 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:29.305115 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:29.305693 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:29.306747 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:29.307093 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:29.307169 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:29.308224 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:29.308229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:29.308232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:53:29.308795 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:29.308829 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:53:29.308849 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:29.308937 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:29.309056 (+ 119us) server_negotiation.cc:299] Negotiation successful
0809 07:53:29.309130 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 07:53:29.312120 1203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d76c5f1b29194efba267183c5caa7c6e: Generated new TSK 0
I20260809 07:53:29.312835 1203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:53:29.319887 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 1070
2026-08-09T07:53:29Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1484 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:53:29 dist-test-slave-vpvm krb5kdc[1218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:53:29 dist-test-slave-vpvm krb5kdc[1218](info): set up 2 sockets
Aug 09 07:53:29 dist-test-slave-vpvm krb5kdc[1218](info): commencing operation
krb5kdc: starting...
W20260809 07:53:30.455837 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:53:30 dist-test-slave-vpvm krb5kdc[1218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:53:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:53:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:53:30.570672 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:34109
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:34109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:30.680358 1242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:30.680620 1242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:30.680683 1242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:30.684688 1242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:30.684752 1242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:30.684784 1242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:30.684811 1242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:30.684845 1242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:53:30.689925 1242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:34109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:34109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:30.690932 1242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:30.691885 1242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:30.699887 1254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:30.699913 1252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:30.700066 1242 server_base.cc:1034] running on GCE node
W20260809 07:53:30.699955 1256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:30.700670 1242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:30.701632 1242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:30.702788 1242 hybrid_clock.cc:648] HybridClock initialized: now 1786262010702774 us; error 27 us; skew 500 ppm
Aug 09 07:53:30 dist-test-slave-vpvm krb5kdc[1218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262010, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:30.705351 1242 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:53:30.706238 1242 webserver.cc:460] Webserver started at http://127.28.112.62:36339/ using document root <none> and password file <none>
I20260809 07:53:30.706719 1242 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:30.706773 1242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:30.706939 1242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:30.708811 1242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "b1dc31d9c5b9489ebc4d559687cfc4fd"
format_stamp: "Formatted at 2026-08-09 07:53:30 on dist-test-slave-vpvm"
server_key: "b339f90c1485f44d9c3cc0416d949491"
server_key_iv: "829da007f3742b260444e295ab072e4b"
server_key_version: "encryptionkey@0"
I20260809 07:53:30.709307 1242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "b1dc31d9c5b9489ebc4d559687cfc4fd"
format_stamp: "Formatted at 2026-08-09 07:53:30 on dist-test-slave-vpvm"
server_key: "b339f90c1485f44d9c3cc0416d949491"
server_key_iv: "829da007f3742b260444e295ab072e4b"
server_key_version: "encryptionkey@0"
I20260809 07:53:30.712651 1242 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:53:30.714887 1266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:30.716152 1242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:30.716274 1242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "b1dc31d9c5b9489ebc4d559687cfc4fd"
format_stamp: "Formatted at 2026-08-09 07:53:30 on dist-test-slave-vpvm"
server_key: "b339f90c1485f44d9c3cc0416d949491"
server_key_iv: "829da007f3742b260444e295ab072e4b"
server_key_version: "encryptionkey@0"
I20260809 07:53:30.716365 1242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:30.735800 1242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:30.738382 1242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:30.738548 1242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:30.744961 1242 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:34109
I20260809 07:53:30.745209 1338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:34109 every 8 connection(s)
I20260809 07:53:30.745849 1242 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:53:30.748522 1339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:30.753628 1339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd: Bootstrap starting.
I20260809 07:53:30.755987 1339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:30.756048 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 1242
I20260809 07:53:30.756178 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:53:30.756472 29120 external_mini_cluster.cc:1428] Setting key 9913d3263eafde67b616ea6b47bebebb
I20260809 07:53:30.756866 1339 log.cc:824] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:30.758826 1339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd: No bootstrap required, opened a new log
I20260809 07:53:30.762456 1339 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1dc31d9c5b9489ebc4d559687cfc4fd" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 34109 } }
I20260809 07:53:30.762661 1339 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:30.762722 1339 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1dc31d9c5b9489ebc4d559687cfc4fd, State: Initialized, Role: FOLLOWER
I20260809 07:53:30.763096 1339 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1dc31d9c5b9489ebc4d559687cfc4fd" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 34109 } }
I20260809 07:53:30.763236 1339 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:53:30.763294 1339 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:53:30.763383 1339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:30.764217 1339 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1dc31d9c5b9489ebc4d559687cfc4fd" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 34109 } }
Aug 09 07:53:30 dist-test-slave-vpvm krb5kdc[1218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:30.764575 1339 leader_election.cc:302] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1dc31d9c5b9489ebc4d559687cfc4fd; no voters:
I20260809 07:53:30.764892 1339 leader_election.cc:288] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:53:30.764968 1344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:30.765178 1344 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd [term 1 LEADER]: Becoming Leader. State: Replica: b1dc31d9c5b9489ebc4d559687cfc4fd, State: Running, Role: LEADER
I20260809 07:53:30.765539 1344 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1dc31d9c5b9489ebc4d559687cfc4fd" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 34109 } }
I20260809 07:53:30.766207 1339 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:30.767093 1346 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1dc31d9c5b9489ebc4d559687cfc4fd. Latest consensus state: current_term: 1 leader_uuid: "b1dc31d9c5b9489ebc4d559687cfc4fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1dc31d9c5b9489ebc4d559687cfc4fd" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 34109 } } }
I20260809 07:53:30.767211 1346 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:30.767519 1345 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1dc31d9c5b9489ebc4d559687cfc4fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1dc31d9c5b9489ebc4d559687cfc4fd" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 34109 } } }
I20260809 07:53:30.767598 1345 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:30.770565 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:30.757500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45806 (local address 127.28.112.62:34109)
0809 07:53:30.757853 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:30.757869 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:30.757913 (+ 44us) server_negotiation.cc:407] Connection header received
0809 07:53:30.758550 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:30.758568 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:30.758856 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:30.759174 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:30.760032 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:30.761052 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:30.761882 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:30.762406 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:30.762558 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:30.765292 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:30.765299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:30.765314 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:30.765344 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:30.767309 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:30.767853 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:30.767860 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:30.767866 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:30.768011 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:30.768344 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:30.768348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:30.768350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:30.768864 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:30.769044 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:30.769367 (+ 323us) server_negotiation.cc:299] Negotiation successful
0809 07:53:30.769667 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":89,"threads_started":1}
W20260809 07:53:30.770870 1362 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:53:30.770954 1362 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:53:30.771013 1356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:53:30.771735 1356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:30.776804 1356 catalog_manager.cc:1347] Generated new cluster ID: bc77b4f1644244169de3ebaf868a4c5a
I20260809 07:53:30.776865 1356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:30.783846 1356 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:53:30.784623 1356 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:30.796712 1356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1dc31d9c5b9489ebc4d559687cfc4fd: Generated new TSK 0
I20260809 07:53:30.797312 1356 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
I20260809 07:53:30.826843 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:34109
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:30.936163 1371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:30.936427 1371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:30.936482 1371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:30.940418 1371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:30.940466 1371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:30.940538 1371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:53:30.945626 1371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:34109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:30.946593 1371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:30.947564 1371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:30.956779 1381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:30.956785 1380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:30.956797 1385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:30.956946 1371 server_base.cc:1034] running on GCE node
I20260809 07:53:30.957386 1371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:30.957988 1371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:30.959173 1371 hybrid_clock.cc:648] HybridClock initialized: now 1786262010959154 us; error 33 us; skew 500 ppm
Aug 09 07:53:30 dist-test-slave-vpvm krb5kdc[1218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262010, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:30.961726 1371 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:53:30.962580 1371 webserver.cc:460] Webserver started at http://127.28.112.1:40151/ using document root <none> and password file <none>
I20260809 07:53:30.963065 1371 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:30.963122 1371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:30.963333 1371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:30.965191 1371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "b8a0ee028aca469a907aad7e83cd1032"
format_stamp: "Formatted at 2026-08-09 07:53:30 on dist-test-slave-vpvm"
server_key: "ab6f1c7aecee5d31066944c510962b9a"
server_key_iv: "039b4f388b92aaf0e9eb8bddfd2de4c0"
server_key_version: "encryptionkey@0"
I20260809 07:53:30.965679 1371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "b8a0ee028aca469a907aad7e83cd1032"
format_stamp: "Formatted at 2026-08-09 07:53:30 on dist-test-slave-vpvm"
server_key: "ab6f1c7aecee5d31066944c510962b9a"
server_key_iv: "039b4f388b92aaf0e9eb8bddfd2de4c0"
server_key_version: "encryptionkey@0"
I20260809 07:53:30.969225 1371 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:53:30.971521 1395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:30.972653 1371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:30.972752 1371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "b8a0ee028aca469a907aad7e83cd1032"
format_stamp: "Formatted at 2026-08-09 07:53:30 on dist-test-slave-vpvm"
server_key: "ab6f1c7aecee5d31066944c510962b9a"
server_key_iv: "039b4f388b92aaf0e9eb8bddfd2de4c0"
server_key_version: "encryptionkey@0"
I20260809 07:53:30.972849 1371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:30.984717 1371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:30.987425 1371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:30.987571 1371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:30.988060 1371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:30.988801 1371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:30.988848 1371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:30.988885 1371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:30.988904 1371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:30.997232 1371 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:41547
I20260809 07:53:30.997279 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:41547 every 8 connection(s)
I20260809 07:53:30.998108 1371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:53:31.002259 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 1371
I20260809 07:53:31.002381 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:53:31.002665 29120 external_mini_cluster.cc:1428] Setting key 81453650c6c4771b2c436eef3abc01b0
Aug 09 07:53:31 dist-test-slave-vpvm krb5kdc[1218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262010, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:31.012275 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:30.999835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:53879 (local address 127.28.112.62:34109)
0809 07:53:30.999956 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:30.999959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:31.000755 (+ 796us) server_negotiation.cc:407] Connection header received
0809 07:53:31.002152 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:31.002156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:31.002216 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:31.002291 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:31.003613 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:31.004245 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:31.005385 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:31.005720 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:31.005828 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:31.008534 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:31.008537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:31.008539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:31.008564 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:31.010176 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:31.010771 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:31.010773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:31.010775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:31.010848 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:31.011202 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:31.011205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:31.011206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:31.011389 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:31.011482 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:31.012054 (+ 572us) server_negotiation.cc:299] Negotiation successful
0809 07:53:31.012154 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for kudu/127.28.112.2@KRBTEST.COM; defaulting to no policy
I20260809 07:53:31.013406 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:31.000164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:34109 (local address 127.28.112.1:53879)
0809 07:53:31.000611 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:31.000653 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:31.001899 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:31.002482 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:31.002491 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:31.002877 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:31.003477 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:31.003497 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:31.004393 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:31.004396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:31.005265 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:31.005275 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:31.005887 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:31.005892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:31.006128 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:31.006690 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:31.006711 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:31.008348 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:31.010297 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:31.010304 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:31.010320 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:31.010676 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:31.010925 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:31.010928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:31.010930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:31.011099 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:31.011500 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:31.011507 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:31.011821 (+ 314us) client_negotiation.cc:769] Sending connection context
0809 07:53:31.012033 (+ 212us) client_negotiation.cc:240] Negotiation successful
0809 07:53:31.012367 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":115,"threads_started":1}
I20260809 07:53:31.014569 1556 heartbeater.cc:347] Connected to a master server at 127.28.112.62:34109
I20260809 07:53:31.014787 1556 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:31.015281 1556 heartbeater.cc:511] Master 127.28.112.62:34109 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.2@KRBTEST.COM" created.
I20260809 07:53:31.016960 1287 ts_manager.cc:194] Registered new tserver with Master: b8a0ee028aca469a907aad7e83cd1032 (127.28.112.1:41547)
I20260809 07:53:31.018131 1287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:53879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
I20260809 07:53:31.029072 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:34109
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:31.136348 1563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:31.136564 1563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:31.136601 1563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:31.140463 1563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:31.140508 1563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:31.140568 1563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:53:31.145533 1563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:34109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:31.146425 1563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:31.147346 1563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:31.156562 1575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:31.156630 1573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:31.156759 1563 server_base.cc:1034] running on GCE node
W20260809 07:53:31.156584 1577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:31.157279 1563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:31.157859 1563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:31.159024 1563 hybrid_clock.cc:648] HybridClock initialized: now 1786262011158992 us; error 42 us; skew 500 ppm
Aug 09 07:53:31 dist-test-slave-vpvm krb5kdc[1218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262011, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:31.161545 1563 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:53:31.162426 1563 webserver.cc:460] Webserver started at http://127.28.112.2:44585/ using document root <none> and password file <none>
I20260809 07:53:31.162966 1563 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:31.163028 1563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:31.163209 1563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:31.165130 1563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "28fac90dffb94c1998168221f8d78595"
format_stamp: "Formatted at 2026-08-09 07:53:31 on dist-test-slave-vpvm"
server_key: "3f0f472a66512be0e78e3d8d8719f510"
server_key_iv: "dca0085c95b2c9dd2beff8dca738148c"
server_key_version: "encryptionkey@0"
I20260809 07:53:31.165621 1563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "28fac90dffb94c1998168221f8d78595"
format_stamp: "Formatted at 2026-08-09 07:53:31 on dist-test-slave-vpvm"
server_key: "3f0f472a66512be0e78e3d8d8719f510"
server_key_iv: "dca0085c95b2c9dd2beff8dca738148c"
server_key_version: "encryptionkey@0"
I20260809 07:53:31.169124 1563 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:53:31.171367 1589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:31.172487 1563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:31.172605 1563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "28fac90dffb94c1998168221f8d78595"
format_stamp: "Formatted at 2026-08-09 07:53:31 on dist-test-slave-vpvm"
server_key: "3f0f472a66512be0e78e3d8d8719f510"
server_key_iv: "dca0085c95b2c9dd2beff8dca738148c"
server_key_version: "encryptionkey@0"
I20260809 07:53:31.172691 1563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:31.189774 1563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:31.192878 1563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:31.193048 1563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:31.193607 1563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:31.194463 1563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:31.194519 1563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:31.194582 1563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:31.194610 1563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:31.202939 1563 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:38685
I20260809 07:53:31.202971 1760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:38685 every 8 connection(s)
I20260809 07:53:31.203799 1563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:53:31.204627 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 1563
I20260809 07:53:31.204739 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:53:31.205001 29120 external_mini_cluster.cc:1428] Setting key 15256d004c7b01cacda417a7ad33df3a
Aug 09 07:53:31 dist-test-slave-vpvm krb5kdc[1218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262011, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.112.3@KRBTEST.COM; defaulting to no policy
I20260809 07:53:31.217367 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:31.205564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:48009 (local address 127.28.112.62:34109)
0809 07:53:31.205716 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:31.205731 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:31.206635 (+ 904us) server_negotiation.cc:407] Connection header received
0809 07:53:31.208315 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:31.208318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:31.208364 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:31.208432 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:31.209621 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:31.210253 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:31.211333 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:31.211563 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:31.211660 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:31.213989 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:31.213994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:31.213997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:31.214025 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:31.215396 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:31.215853 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:31.215856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:31.215858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:31.215933 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:31.216328 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:31.216332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:31.216334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:31.216518 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:31.216606 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:31.217080 (+ 474us) server_negotiation.cc:299] Negotiation successful
0809 07:53:31.217205 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.3@KRBTEST.COM" created.
I20260809 07:53:31.218418 1764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:31.206007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:34109 (local address 127.28.112.2:48009)
0809 07:53:31.206469 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:31.206526 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:31.208094 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:31.208653 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:31.208663 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:31.209005 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:31.209467 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:31.209481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:31.210350 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:31.210353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:31.211211 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:31.211221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:31.211797 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:31.211801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:31.211938 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:31.212449 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:31.212466 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:31.213803 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:31.215495 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:31.215501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:31.215516 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:31.215755 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:31.216024 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:31.216062 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:31.216064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:31.216232 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:31.216620 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:31.216626 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:31.216855 (+ 229us) client_negotiation.cc:769] Sending connection context
0809 07:53:31.217026 (+ 171us) client_negotiation.cc:240] Negotiation successful
0809 07:53:31.217417 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":114,"threads_started":1}
I20260809 07:53:31.219493 1761 heartbeater.cc:347] Connected to a master server at 127.28.112.62:34109
I20260809 07:53:31.219720 1761 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:31.220168 1761 heartbeater.cc:511] Master 127.28.112.62:34109 requested a full tablet report, sending...
I20260809 07:53:31.221292 1287 ts_manager.cc:194] Registered new tserver with Master: 28fac90dffb94c1998168221f8d78595 (127.28.112.2:38685)
I20260809 07:53:31.222066 1287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:48009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
I20260809 07:53:31.232088 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:34109
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:31.341110 1767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:31.341369 1767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:31.341416 1767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:31.345374 1767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:31.345431 1767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:31.345487 1767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:53:31.350567 1767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:34109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:31.351506 1767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:31.352481 1767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:31.361930 1775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:31.362010 1777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:31.362180 1767 server_base.cc:1034] running on GCE node
W20260809 07:53:31.362253 1774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:31.362627 1767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:31.363248 1767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:31.364413 1767 hybrid_clock.cc:648] HybridClock initialized: now 1786262011364394 us; error 30 us; skew 500 ppm
Aug 09 07:53:31 dist-test-slave-vpvm krb5kdc[1218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262011, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:31.366940 1767 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:53:31.367767 1767 webserver.cc:460] Webserver started at http://127.28.112.3:45583/ using document root <none> and password file <none>
I20260809 07:53:31.368265 1767 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:31.368330 1767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:31.368517 1767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:31.370405 1767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "f936d5f8072242dfb535581c9fe91383"
format_stamp: "Formatted at 2026-08-09 07:53:31 on dist-test-slave-vpvm"
server_key: "b2f99d4e1c7cc39b29aaeda2c10cbe7f"
server_key_iv: "baebf46d350f22394bdc2a8a3d9f1533"
server_key_version: "encryptionkey@0"
I20260809 07:53:31.370908 1767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "f936d5f8072242dfb535581c9fe91383"
format_stamp: "Formatted at 2026-08-09 07:53:31 on dist-test-slave-vpvm"
server_key: "b2f99d4e1c7cc39b29aaeda2c10cbe7f"
server_key_iv: "baebf46d350f22394bdc2a8a3d9f1533"
server_key_version: "encryptionkey@0"
I20260809 07:53:31.374262 1767 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:53:31.376529 1785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:31.377682 1767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:31.377785 1767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "f936d5f8072242dfb535581c9fe91383"
format_stamp: "Formatted at 2026-08-09 07:53:31 on dist-test-slave-vpvm"
server_key: "b2f99d4e1c7cc39b29aaeda2c10cbe7f"
server_key_iv: "baebf46d350f22394bdc2a8a3d9f1533"
server_key_version: "encryptionkey@0"
I20260809 07:53:31.377867 1767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:31.407207 1767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:31.409914 1767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:31.410132 1767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:31.410718 1767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:31.411547 1767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:31.411594 1767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:31.411646 1767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:31.411671 1767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:31.419972 1767 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:33969
I20260809 07:53:31.420027 1951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:33969 every 8 connection(s)
I20260809 07:53:31.420889 1767 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:53:31.427934 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 1767
I20260809 07:53:31.428038 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:53:31.428349 29120 external_mini_cluster.cc:1428] Setting key 98d3b7643656e9b10380c788eb269455
Aug 09 07:53:31 dist-test-slave-vpvm krb5kdc[1218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262011, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:31.434270 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:31.422694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:44247 (local address 127.28.112.62:34109)
0809 07:53:31.422814 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:31.422817 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:31.423663 (+ 846us) server_negotiation.cc:407] Connection header received
0809 07:53:31.425084 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:31.425087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:31.425157 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:31.425230 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:31.426423 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:31.426957 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:31.428062 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:31.428376 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:31.428494 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:31.431082 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:31.431086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:31.431089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:31.431115 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:31.432358 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:31.432838 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:31.432842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:31.432844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:31.432920 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:31.433241 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:31.433244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:31.433246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:31.433418 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:31.433504 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:31.433998 (+ 494us) server_negotiation.cc:299] Negotiation successful
0809 07:53:31.434114 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 07:53:31.435007 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:31.423000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:34109 (local address 127.28.112.3:44247)
0809 07:53:31.423505 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:31.423549 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:31.424878 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:31.425409 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:31.425416 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:31.425769 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:31.426288 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:31.426302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:31.427079 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:31.427082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:31.427928 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:31.427938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:31.428533 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:31.428537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:31.428711 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:31.429329 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:31.429345 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:31.430931 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:31.432470 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:31.432476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:31.432491 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:31.432743 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:31.433003 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:31.433006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:31.433008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:31.433151 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:31.433518 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:31.433525 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:31.433753 (+ 228us) client_negotiation.cc:769] Sending connection context
0809 07:53:31.433915 (+ 162us) client_negotiation.cc:240] Negotiation successful
0809 07:53:31.434251 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":131,"threads_started":1}
I20260809 07:53:31.435999 1953 heartbeater.cc:347] Connected to a master server at 127.28.112.62:34109
I20260809 07:53:31.436226 1953 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:31.436662 1953 heartbeater.cc:511] Master 127.28.112.62:34109 requested a full tablet report, sending...
I20260809 07:53:31.437569 1287 ts_manager.cc:194] Registered new tserver with Master: f936d5f8072242dfb535581c9fe91383 (127.28.112.3:33969)
I20260809 07:53:31.438145 1287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:44247
I20260809 07:53:31.441895 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 07:53:31 dist-test-slave-vpvm krb5kdc[1218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262011, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 09 07:53:31 dist-test-slave-vpvm krb5kdc[1218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262011, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:31.466735 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:31.459070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45808 (local address 127.28.112.62:34109)
0809 07:53:31.459192 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:31.459196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:31.459326 (+ 130us) server_negotiation.cc:407] Connection header received
0809 07:53:31.459444 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:31.459452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:31.459496 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:31.459572 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:31.460202 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:31.460734 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:31.461517 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:31.461730 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:31.461806 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:31.464036 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:31.464040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:31.464041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:31.464066 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:31.465216 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:31.465571 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:31.465573 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:31.465575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:31.465645 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:31.465827 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:31.465829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:31.465830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:31.466124 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:31.466243 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:31.466484 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 07:53:31.466584 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260809 07:53:31.467212 1287 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45808
I20260809 07:53:31.468072 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 1371
I20260809 07:53:31.487644 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 1563
I20260809 07:53:31.499714 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 1767
I20260809 07:53:31.511160 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 1242
2026-08-09T07:53:31Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2189 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:53:31 dist-test-slave-vpvm krb5kdc[1968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:53:31 dist-test-slave-vpvm krb5kdc[1968](info): set up 2 sockets
Aug 09 07:53:31 dist-test-slave-vpvm krb5kdc[1968](info): commencing operation
krb5kdc: starting...
W20260809 07:53:32.655524 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:53:32 dist-test-slave-vpvm krb5kdc[1968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:53:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:53:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:53:32.769153 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:38093
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:38093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:32.876083 1986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:32.876314 1986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:32.876356 1986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:32.880237 1986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:32.880288 1986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:32.880317 1986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:32.880347 1986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:32.880374 1986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:53:32.885552 1986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:38093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:38093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:32.886567 1986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:32.887549 1986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:32.895591 2001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:32.895555 1997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:32.895591 1998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:32.895905 1986 server_base.cc:1034] running on GCE node
I20260809 07:53:32.896426 1986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:32.897435 1986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:32.898586 1986 hybrid_clock.cc:648] HybridClock initialized: now 1786262012898558 us; error 39 us; skew 500 ppm
Aug 09 07:53:32 dist-test-slave-vpvm krb5kdc[1968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262012, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:32.901093 1986 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:53:32.901933 1986 webserver.cc:460] Webserver started at http://127.28.112.62:37027/ using document root <none> and password file <none>
I20260809 07:53:32.902472 1986 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:32.902559 1986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:32.902776 1986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:32.904708 1986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "73aa94d44ac4474591b438c74e65d7a4"
format_stamp: "Formatted at 2026-08-09 07:53:32 on dist-test-slave-vpvm"
server_key: "2d6063179f33ecb55c1756224a1441e2"
server_key_iv: "2bb55f63460bd62d189e6a9c1e3e1f47"
server_key_version: "encryptionkey@0"
I20260809 07:53:32.905218 1986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "73aa94d44ac4474591b438c74e65d7a4"
format_stamp: "Formatted at 2026-08-09 07:53:32 on dist-test-slave-vpvm"
server_key: "2d6063179f33ecb55c1756224a1441e2"
server_key_iv: "2bb55f63460bd62d189e6a9c1e3e1f47"
server_key_version: "encryptionkey@0"
I20260809 07:53:32.908644 1986 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260809 07:53:32.910872 2014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:32.911983 1986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 07:53:32.912093 1986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "73aa94d44ac4474591b438c74e65d7a4"
format_stamp: "Formatted at 2026-08-09 07:53:32 on dist-test-slave-vpvm"
server_key: "2d6063179f33ecb55c1756224a1441e2"
server_key_iv: "2bb55f63460bd62d189e6a9c1e3e1f47"
server_key_version: "encryptionkey@0"
I20260809 07:53:32.912179 1986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:32.927408 1986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:32.929781 1986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:32.929937 1986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:32.936293 1986 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:38093
I20260809 07:53:32.936328 2097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:38093 every 8 connection(s)
I20260809 07:53:32.937175 1986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:53:32.939879 2100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:32.944363 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 1986
I20260809 07:53:32.944482 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:53:32.944717 29120 external_mini_cluster.cc:1428] Setting key 074a493db519c69f763d7c08603e6bc8
I20260809 07:53:32.945254 2100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4: Bootstrap starting.
I20260809 07:53:32.947522 2100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:32.948411 2100 log.cc:824] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:32.950377 2100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4: No bootstrap required, opened a new log
Aug 09 07:53:32 dist-test-slave-vpvm krb5kdc[1968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:32.954437 2100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73aa94d44ac4474591b438c74e65d7a4" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38093 } }
I20260809 07:53:32.954674 2100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:32.954733 2100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73aa94d44ac4474591b438c74e65d7a4, State: Initialized, Role: FOLLOWER
I20260809 07:53:32.955188 2100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73aa94d44ac4474591b438c74e65d7a4" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38093 } }
I20260809 07:53:32.955348 2100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:53:32.955425 2100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:53:32.955519 2100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:32.956451 2100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73aa94d44ac4474591b438c74e65d7a4" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38093 } }
I20260809 07:53:32.956799 2100 leader_election.cc:302] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73aa94d44ac4474591b438c74e65d7a4; no voters:
I20260809 07:53:32.957131 2100 leader_election.cc:288] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:53:32.957218 2105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:32.957419 2105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4 [term 1 LEADER]: Becoming Leader. State: Replica: 73aa94d44ac4474591b438c74e65d7a4, State: Running, Role: LEADER
I20260809 07:53:32.957728 2105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73aa94d44ac4474591b438c74e65d7a4" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38093 } }
I20260809 07:53:32.958415 2100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:32.958834 2105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73aa94d44ac4474591b438c74e65d7a4. Latest consensus state: current_term: 1 leader_uuid: "73aa94d44ac4474591b438c74e65d7a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73aa94d44ac4474591b438c74e65d7a4" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38093 } } }
I20260809 07:53:32.958974 2105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:32.959358 2106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73aa94d44ac4474591b438c74e65d7a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73aa94d44ac4474591b438c74e65d7a4" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38093 } } }
I20260809 07:53:32.959447 2112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:53:32.959534 2106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:32.959551 2103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:32.945930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43088 (local address 127.28.112.62:38093)
0809 07:53:32.946335 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:32.946348 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:32.946383 (+ 35us) server_negotiation.cc:407] Connection header received
0809 07:53:32.946959 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:32.946974 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:32.947276 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:32.947540 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:32.948410 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:32.949365 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:32.950496 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:32.950961 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:32.951080 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:32.953813 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:32.953821 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:32.953834 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:32.953862 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:32.955971 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:32.956409 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:32.956416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:32.956422 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:32.956568 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:32.956835 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:32.956838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:32.956840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:32.957361 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:32.958123 (+ 762us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:32.958284 (+ 161us) server_negotiation.cc:299] Negotiation successful
0809 07:53:32.958563 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":88,"threads_started":1}
I20260809 07:53:32.962564 2112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:32.967717 2112 catalog_manager.cc:1347] Generated new cluster ID: cd3085165aad4fee9bf0e1502ec5aa42
I20260809 07:53:32.967777 2112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:32.977557 2112 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:53:32.978870 2112 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:32.985980 2112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73aa94d44ac4474591b438c74e65d7a4: Generated new TSK 0
I20260809 07:53:32.986706 2112 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
I20260809 07:53:33.016963 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:38093
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:33.126178 2132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:33.126405 2132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:33.126454 2132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:33.130307 2132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:33.130354 2132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:33.130406 2132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:53:33.135285 2132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:38093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:33.136154 2132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:33.137032 2132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:33.145889 2142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:33.145922 2145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:33.145928 2143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:33.145989 2132 server_base.cc:1034] running on GCE node
I20260809 07:53:33.146585 2132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:33.147169 2132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:33.148322 2132 hybrid_clock.cc:648] HybridClock initialized: now 1786262013148302 us; error 31 us; skew 500 ppm
Aug 09 07:53:33 dist-test-slave-vpvm krb5kdc[1968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262013, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:33.150892 2132 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:53:33.151675 2132 webserver.cc:460] Webserver started at http://127.28.112.1:46685/ using document root <none> and password file <none>
I20260809 07:53:33.152215 2132 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:33.152278 2132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:33.152458 2132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:33.154407 2132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "88099b9e62624602b42c4eac2517f458"
format_stamp: "Formatted at 2026-08-09 07:53:33 on dist-test-slave-vpvm"
server_key: "d482b1478ef7adcc0bfb6df3c9e1db0a"
server_key_iv: "2b8b4b137d96cce7a8dd62d091f03b9f"
server_key_version: "encryptionkey@0"
I20260809 07:53:33.154932 2132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "88099b9e62624602b42c4eac2517f458"
format_stamp: "Formatted at 2026-08-09 07:53:33 on dist-test-slave-vpvm"
server_key: "d482b1478ef7adcc0bfb6df3c9e1db0a"
server_key_iv: "2b8b4b137d96cce7a8dd62d091f03b9f"
server_key_version: "encryptionkey@0"
I20260809 07:53:33.158419 2132 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 07:53:33.160744 2153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:33.161846 2132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:33.161994 2132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "88099b9e62624602b42c4eac2517f458"
format_stamp: "Formatted at 2026-08-09 07:53:33 on dist-test-slave-vpvm"
server_key: "d482b1478ef7adcc0bfb6df3c9e1db0a"
server_key_iv: "2b8b4b137d96cce7a8dd62d091f03b9f"
server_key_version: "encryptionkey@0"
I20260809 07:53:33.162086 2132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:33.175943 2132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:33.178719 2132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:33.178879 2132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:33.179395 2132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:33.180222 2132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:33.180276 2132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:33.180328 2132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:33.180357 2132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:33.188549 2132 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:39611
I20260809 07:53:33.188587 2320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:39611 every 8 connection(s)
I20260809 07:53:33.189388 2132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:53:33.192777 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 2132
I20260809 07:53:33.192894 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:53:33.193197 29120 external_mini_cluster.cc:1428] Setting key fea89b6da4dd87e621d147d9e3cbf120
Aug 09 07:53:33 dist-test-slave-vpvm krb5kdc[1968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262013, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:33.203454 2103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:33.191085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:58085 (local address 127.28.112.62:38093)
0809 07:53:33.191215 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:33.191219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:33.191972 (+ 753us) server_negotiation.cc:407] Connection header received
0809 07:53:33.193577 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:33.193581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:33.193642 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:33.193728 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:33.195109 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:33.195791 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:33.197152 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:33.197479 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:33.197586 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:33.200107 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:33.200110 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:33.200112 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:33.200135 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:33.201387 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:33.201998 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:33.202001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:33.202002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:33.202073 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:33.202419 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:33.202422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:33.202424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:33.202576 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:33.202643 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:33.203240 (+ 597us) server_negotiation.cc:299] Negotiation successful
0809 07:53:33.203331 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.28.112.2@KRBTEST.COM; defaulting to no policy
I20260809 07:53:33.204344 2324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:33.191408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38093 (local address 127.28.112.1:58085)
0809 07:53:33.191831 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:33.191870 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:33.193343 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:33.193917 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:33.193926 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:33.194317 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:33.194947 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:33.194968 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:33.195926 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:33.195930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:33.197034 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:33.197045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:33.197608 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:33.197610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:33.197747 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:33.198267 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:33.198283 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:33.199948 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:33.201503 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:33.201510 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:33.201525 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:33.201881 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:33.202155 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:33.202158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:33.202160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:33.202336 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:33.202679 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:33.202687 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:33.202993 (+ 306us) client_negotiation.cc:769] Sending connection context
0809 07:53:33.203218 (+ 225us) client_negotiation.cc:240] Negotiation successful
0809 07:53:33.203530 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":114,"threads_started":1}
I20260809 07:53:33.205400 2322 heartbeater.cc:347] Connected to a master server at 127.28.112.62:38093
I20260809 07:53:33.205628 2322 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.2@KRBTEST.COM" created.
I20260809 07:53:33.206148 2322 heartbeater.cc:511] Master 127.28.112.62:38093 requested a full tablet report, sending...
I20260809 07:53:33.207700 2046 ts_manager.cc:194] Registered new tserver with Master: 88099b9e62624602b42c4eac2517f458 (127.28.112.1:39611)
I20260809 07:53:33.209227 2046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:58085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
I20260809 07:53:33.221972 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:38093
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:33.330021 2330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:33.330267 2330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:33.330307 2330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:33.334295 2330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:33.334348 2330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:33.334403 2330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:53:33.339454 2330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:38093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:33.340371 2330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:33.341334 2330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:33.350415 2337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:33.350456 2339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:33.350455 2341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:33.350672 2330 server_base.cc:1034] running on GCE node
I20260809 07:53:33.351104 2330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:33.351738 2330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:33.352895 2330 hybrid_clock.cc:648] HybridClock initialized: now 1786262013352878 us; error 28 us; skew 500 ppm
Aug 09 07:53:33 dist-test-slave-vpvm krb5kdc[1968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262013, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:33.355520 2330 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:53:33.356381 2330 webserver.cc:460] Webserver started at http://127.28.112.2:37379/ using document root <none> and password file <none>
I20260809 07:53:33.356915 2330 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:33.356988 2330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:33.357194 2330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:33.359299 2330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "be2876c520af493b9e60e1c04143515f"
format_stamp: "Formatted at 2026-08-09 07:53:33 on dist-test-slave-vpvm"
server_key: "84510f9770da8a54827851796cc96991"
server_key_iv: "fec18cef2cf9ac8df556868410ebef2e"
server_key_version: "encryptionkey@0"
I20260809 07:53:33.359867 2330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "be2876c520af493b9e60e1c04143515f"
format_stamp: "Formatted at 2026-08-09 07:53:33 on dist-test-slave-vpvm"
server_key: "84510f9770da8a54827851796cc96991"
server_key_iv: "fec18cef2cf9ac8df556868410ebef2e"
server_key_version: "encryptionkey@0"
I20260809 07:53:33.363446 2330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 07:53:33.365787 2354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:33.366983 2330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:33.367094 2330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "be2876c520af493b9e60e1c04143515f"
format_stamp: "Formatted at 2026-08-09 07:53:33 on dist-test-slave-vpvm"
server_key: "84510f9770da8a54827851796cc96991"
server_key_iv: "fec18cef2cf9ac8df556868410ebef2e"
server_key_version: "encryptionkey@0"
I20260809 07:53:33.367189 2330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:33.395686 2330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:33.398732 2330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:33.398902 2330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:33.399492 2330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:33.400326 2330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:33.400375 2330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:33.400431 2330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:33.400458 2330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:33.409150 2330 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:46461
I20260809 07:53:33.409196 2525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:46461 every 8 connection(s)
I20260809 07:53:33.410076 2330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:53:33.418105 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 2330
I20260809 07:53:33.418207 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:53:33.418469 29120 external_mini_cluster.cc:1428] Setting key ae7b25bd5af0a07ea8527b5346e343bb
Aug 09 07:53:33 dist-test-slave-vpvm krb5kdc[1968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262013, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:33.422647 2103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:33.411838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:41135 (local address 127.28.112.62:38093)
0809 07:53:33.411975 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:33.411978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:33.412762 (+ 784us) server_negotiation.cc:407] Connection header received
0809 07:53:33.414090 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:33.414093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:33.414139 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:33.414209 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:33.415275 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:33.415828 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:33.416938 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:33.417159 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:33.417234 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:33.419476 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:33.419479 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:33.419481 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:33.419504 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:33.420781 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:33.421237 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:33.421241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:33.421243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:33.421317 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:33.421618 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:33.421622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:33.421623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:33.421793 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:33.421876 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:33.422372 (+ 496us) server_negotiation.cc:299] Negotiation successful
0809 07:53:33.422489 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 07:53:33.423550 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:33.412178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38093 (local address 127.28.112.2:41135)
0809 07:53:33.412610 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:33.412661 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:33.413872 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:33.414359 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:33.414366 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:33.414704 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:33.415169 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:33.415182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:33.415927 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:33.415930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:33.416848 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:33.416858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:33.417267 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:33.417269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:33.417407 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:33.417866 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:33.417881 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:33.419327 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:33.420885 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:33.420891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:33.420904 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:33.421146 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:33.421396 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:33.421399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:33.421401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:33.421526 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:33.421888 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:33.421894 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:33.422157 (+ 263us) client_negotiation.cc:769] Sending connection context
0809 07:53:33.422315 (+ 158us) client_negotiation.cc:240] Negotiation successful
0809 07:53:33.422620 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
I20260809 07:53:33.424665 2528 heartbeater.cc:347] Connected to a master server at 127.28.112.62:38093
I20260809 07:53:33.424896 2528 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:33.425351 2528 heartbeater.cc:511] Master 127.28.112.62:38093 requested a full tablet report, sending...
I20260809 07:53:33.426381 2046 ts_manager.cc:194] Registered new tserver with Master: be2876c520af493b9e60e1c04143515f (127.28.112.2:46461)
I20260809 07:53:33.426923 2046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:41135
WARNING: no policy specified for kudu/127.28.112.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
I20260809 07:53:33.444504 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:38093
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:33.553061 2534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:33.553313 2534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:33.553370 2534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:33.557785 2534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:33.557848 2534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:33.557915 2534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:53:33.563391 2534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:38093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:33.564353 2534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:33.565356 2534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:33.574389 2545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:33.574398 2542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:33.574465 2547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:33.574666 2534 server_base.cc:1034] running on GCE node
I20260809 07:53:33.575050 2534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:33.575667 2534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:33.576854 2534 hybrid_clock.cc:648] HybridClock initialized: now 1786262013576822 us; error 44 us; skew 500 ppm
Aug 09 07:53:33 dist-test-slave-vpvm krb5kdc[1968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262013, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:33.579733 2534 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:53:33.580534 2534 webserver.cc:460] Webserver started at http://127.28.112.3:35367/ using document root <none> and password file <none>
I20260809 07:53:33.581045 2534 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:33.581110 2534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:33.581312 2534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:33.583359 2534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "f3d1958994a84565b39315a9fa0b005d"
format_stamp: "Formatted at 2026-08-09 07:53:33 on dist-test-slave-vpvm"
server_key: "e612d8d34c260e0345bb345f66515687"
server_key_iv: "abd8ba72611cc996e1801f02e06fa08e"
server_key_version: "encryptionkey@0"
I20260809 07:53:33.583881 2534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "f3d1958994a84565b39315a9fa0b005d"
format_stamp: "Formatted at 2026-08-09 07:53:33 on dist-test-slave-vpvm"
server_key: "e612d8d34c260e0345bb345f66515687"
server_key_iv: "abd8ba72611cc996e1801f02e06fa08e"
server_key_version: "encryptionkey@0"
I20260809 07:53:33.587337 2534 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:53:33.589561 2558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:33.590684 2534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:33.590793 2534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "f3d1958994a84565b39315a9fa0b005d"
format_stamp: "Formatted at 2026-08-09 07:53:33 on dist-test-slave-vpvm"
server_key: "e612d8d34c260e0345bb345f66515687"
server_key_iv: "abd8ba72611cc996e1801f02e06fa08e"
server_key_version: "encryptionkey@0"
I20260809 07:53:33.590879 2534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:33.600159 2534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:33.602665 2534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:33.602824 2534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:33.603371 2534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:33.604151 2534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:33.604199 2534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:33.604250 2534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:33.604278 2534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:33.612394 2534 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:46257
I20260809 07:53:33.612426 2729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:46257 every 8 connection(s)
I20260809 07:53:33.613206 2534 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:53:33.619856 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 2534
I20260809 07:53:33.619966 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:53:33.620208 29120 external_mini_cluster.cc:1428] Setting key cc38f2f9660c24296f911e754c7b7cad
Aug 09 07:53:33 dist-test-slave-vpvm krb5kdc[1968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262013, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:33.625699 2103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:33.614962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:52769 (local address 127.28.112.62:38093)
0809 07:53:33.615085 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:33.615088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:33.615898 (+ 810us) server_negotiation.cc:407] Connection header received
0809 07:53:33.617256 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:33.617259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:33.617318 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:33.617398 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:33.618568 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:33.619093 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:33.620239 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:33.620493 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:33.620603 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:33.622982 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:33.622985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:33.622987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:33.623010 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:33.624130 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:33.624568 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:33.624570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:33.624572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:33.624639 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:33.624880 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:33.624882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:33.624883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:33.625045 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:33.625113 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:33.625493 (+ 380us) server_negotiation.cc:299] Negotiation successful
0809 07:53:33.625584 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 07:53:33.626454 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:33.615312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38093 (local address 127.28.112.3:52769)
0809 07:53:33.615750 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:33.615790 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:33.617046 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:33.617592 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:33.617600 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:33.617928 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:33.618429 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:33.618443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:33.619210 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:33.619213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:33.620127 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:33.620137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:33.620660 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:33.620666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:33.620850 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:33.621396 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:33.621412 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:33.622834 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:33.624225 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:33.624230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:33.624240 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:33.624494 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:33.624699 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:33.624701 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:33.624703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:33.624819 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:33.625112 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:33.625116 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:33.625321 (+ 205us) client_negotiation.cc:769] Sending connection context
0809 07:53:33.625462 (+ 141us) client_negotiation.cc:240] Negotiation successful
0809 07:53:33.625696 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":103,"threads_started":1}
I20260809 07:53:33.627439 2730 heartbeater.cc:347] Connected to a master server at 127.28.112.62:38093
I20260809 07:53:33.627646 2730 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:33.628072 2730 heartbeater.cc:511] Master 127.28.112.62:38093 requested a full tablet report, sending...
I20260809 07:53:33.628965 2046 ts_manager.cc:194] Registered new tserver with Master: f3d1958994a84565b39315a9fa0b005d (127.28.112.3:46257)
I20260809 07:53:33.629483 2046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:52769
I20260809 07:53:33.633574 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 07:53:33 dist-test-slave-vpvm krb5kdc[1968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 07:53:33 dist-test-slave-vpvm krb5kdc[1968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.1@KRBTEST.COM
I20260809 07:53:33.657994 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:33.649758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59070 (local address 127.28.112.1:39611)
0809 07:53:33.650123 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:33.650132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:33.650153 (+ 21us) server_negotiation.cc:407] Connection header received
0809 07:53:33.650198 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:33.650203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:33.650356 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:33.650467 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:33.651047 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:33.651775 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:33.652589 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:33.652966 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:33.653059 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:33.655200 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:33.655209 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:33.655222 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:33.655247 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:33.656525 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:33.656894 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:33.656898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:33.656899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:33.656990 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:33.657178 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:33.657180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:33.657181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:33.657388 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:33.657536 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:33.657669 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 07:53:33.657787 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":141,"threads_started":1}
Aug 09 07:53:33 dist-test-slave-vpvm krb5kdc[1968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:33.668145 2103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:33.660827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43092 (local address 127.28.112.62:38093)
0809 07:53:33.660954 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:33.660958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:33.661077 (+ 119us) server_negotiation.cc:407] Connection header received
0809 07:53:33.661196 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:33.661199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:33.661243 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:33.661301 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:33.661897 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:33.662438 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:33.663182 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:33.663398 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:33.663483 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:33.665625 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:33.665628 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:33.665630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:33.665653 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:33.666810 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:33.667183 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:33.667185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:33.667187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:33.667260 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:33.667444 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:33.667447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:33.667449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:33.667645 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:33.667740 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:33.667889 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 07:53:33.667989 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260809 07:53:33.668507 2046 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43092
I20260809 07:53:33.669045 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 2132
I20260809 07:53:33.677772 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 2330
I20260809 07:53:33.688834 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 2534
I20260809 07:53:33.697398 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 1986
2026-08-09T07:53:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2185 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:53:33 dist-test-slave-vpvm krb5kdc[2745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:53:33 dist-test-slave-vpvm krb5kdc[2745](info): set up 2 sockets
Aug 09 07:53:33 dist-test-slave-vpvm krb5kdc[2745](info): commencing operation
krb5kdc: starting...
W20260809 07:53:34.823673 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:53:34 dist-test-slave-vpvm krb5kdc[2745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:53:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:53:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:53:34.946759 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:43179
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:43179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:35.054466 2763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:35.054710 2763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:35.054767 2763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:35.058744 2763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:35.058802 2763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:35.058827 2763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:35.058849 2763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:35.058879 2763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:53:35.064007 2763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:43179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:43179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:35.065058 2763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:35.066130 2763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 07:53:35.074241 2763 server_base.cc:1034] running on GCE node
W20260809 07:53:35.074199 2769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:35.074259 2771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:35.074199 2773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:35.075016 2763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:35.076117 2763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:35.077303 2763 hybrid_clock.cc:648] HybridClock initialized: now 1786262015077281 us; error 32 us; skew 500 ppm
Aug 09 07:53:35 dist-test-slave-vpvm krb5kdc[2745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:35.080153 2763 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:53:35.080852 2763 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:35.080921 2763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:35.081115 2763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:35.083165 2763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "1d732235db464b01925f0490534bd29a"
format_stamp: "Formatted at 2026-08-09 07:53:35 on dist-test-slave-vpvm"
server_key: "9403ffb70fb6938e1f392ee441b19d20"
server_key_iv: "926a10f9d381f1ad216aa95af246dfd8"
server_key_version: "encryptionkey@0"
I20260809 07:53:35.083710 2763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "1d732235db464b01925f0490534bd29a"
format_stamp: "Formatted at 2026-08-09 07:53:35 on dist-test-slave-vpvm"
server_key: "9403ffb70fb6938e1f392ee441b19d20"
server_key_iv: "926a10f9d381f1ad216aa95af246dfd8"
server_key_version: "encryptionkey@0"
I20260809 07:53:35.088960 2763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 07:53:35.091241 2782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:35.092374 2763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:35.092517 2763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "1d732235db464b01925f0490534bd29a"
format_stamp: "Formatted at 2026-08-09 07:53:35 on dist-test-slave-vpvm"
server_key: "9403ffb70fb6938e1f392ee441b19d20"
server_key_iv: "926a10f9d381f1ad216aa95af246dfd8"
server_key_version: "encryptionkey@0"
I20260809 07:53:35.092609 2763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:35.116586 2763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:35.119652 2763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:35.119828 2763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:35.126240 2763 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:43179
I20260809 07:53:35.126464 2866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:43179 every 8 connection(s)
I20260809 07:53:35.126865 2763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:53:35.129483 2867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:35.132266 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 2763
I20260809 07:53:35.132447 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:53:35.132767 29120 external_mini_cluster.cc:1428] Setting key be29d59d259cb9a4351304ce6b9bb70a
I20260809 07:53:35.135315 2867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a: Bootstrap starting.
I20260809 07:53:35.137790 2867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:35.138689 2867 log.cc:824] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:35.140846 2867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a: No bootstrap required, opened a new log
Aug 09 07:53:35 dist-test-slave-vpvm krb5kdc[2745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:35.143865 2867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d732235db464b01925f0490534bd29a" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43179 } }
I20260809 07:53:35.144078 2867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:35.144137 2867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d732235db464b01925f0490534bd29a, State: Initialized, Role: FOLLOWER
I20260809 07:53:35.144562 2867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d732235db464b01925f0490534bd29a" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43179 } }
I20260809 07:53:35.144704 2867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:53:35.144779 2867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:53:35.144876 2867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:35.145871 2867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d732235db464b01925f0490534bd29a" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43179 } }
I20260809 07:53:35.146246 2867 leader_election.cc:302] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d732235db464b01925f0490534bd29a; no voters:
I20260809 07:53:35.146572 2867 leader_election.cc:288] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:53:35.146662 2876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:35.146886 2876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a [term 1 LEADER]: Becoming Leader. State: Replica: 1d732235db464b01925f0490534bd29a, State: Running, Role: LEADER
I20260809 07:53:35.147217 2876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d732235db464b01925f0490534bd29a" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43179 } }
I20260809 07:53:35.147421 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.134099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39048 (local address 127.28.112.62:43179)
0809 07:53:35.134482 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:35.134497 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:35.134533 (+ 36us) server_negotiation.cc:407] Connection header received
0809 07:53:35.135108 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:35.135124 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:35.135452 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:35.135731 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:35.136583 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.137558 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.138734 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.139371 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.139514 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:35.142191 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:35.142197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:35.142210 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:35.142236 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:35.144318 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:35.144765 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:35.144772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:35.144778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:35.144909 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:35.145214 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:35.145218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:35.145220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:35.145728 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:35.145939 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:35.146145 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 07:53:35.146437 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":100,"threads_started":1}
I20260809 07:53:35.147797 2867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:35.148777 2877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d732235db464b01925f0490534bd29a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d732235db464b01925f0490534bd29a" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43179 } } }
I20260809 07:53:35.148900 2877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:35.149127 2877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d732235db464b01925f0490534bd29a. Latest consensus state: current_term: 1 leader_uuid: "1d732235db464b01925f0490534bd29a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d732235db464b01925f0490534bd29a" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43179 } } }
I20260809 07:53:35.149201 2877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:35.149724 2888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:53:35.152483 2888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:35.157209 2888 catalog_manager.cc:1347] Generated new cluster ID: c0962204a5a346a1ad27478f0e300334
I20260809 07:53:35.157270 2888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:35.169477 2888 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:53:35.170341 2888 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:35.178617 2888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a: Generated new TSK 0
I20260809 07:53:35.179203 2888 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
I20260809 07:53:35.210878 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:43179
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:35.321585 2904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:35.321828 2904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:35.321887 2904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:35.325834 2904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:35.325892 2904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:35.325995 2904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:53:35.331036 2904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:43179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:35.331993 2904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:35.332976 2904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:35.341739 2918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:35.341810 2913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:35.341753 2915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:35.341990 2904 server_base.cc:1034] running on GCE node
I20260809 07:53:35.342432 2904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:35.343048 2904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:35.344214 2904 hybrid_clock.cc:648] HybridClock initialized: now 1786262015344193 us; error 29 us; skew 500 ppm
Aug 09 07:53:35 dist-test-slave-vpvm krb5kdc[2745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:35.346963 2904 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:53:35.347702 2904 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:35.347808 2904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:35.348037 2904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:35.350176 2904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "ac5edf032c4e41299ec075f3c4429626"
format_stamp: "Formatted at 2026-08-09 07:53:35 on dist-test-slave-vpvm"
server_key: "4ecd1a9a0cd99c1a245a1673cb8c31b8"
server_key_iv: "9bafabe8ac3e50a6816141def0bb61d9"
server_key_version: "encryptionkey@0"
I20260809 07:53:35.350726 2904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "ac5edf032c4e41299ec075f3c4429626"
format_stamp: "Formatted at 2026-08-09 07:53:35 on dist-test-slave-vpvm"
server_key: "4ecd1a9a0cd99c1a245a1673cb8c31b8"
server_key_iv: "9bafabe8ac3e50a6816141def0bb61d9"
server_key_version: "encryptionkey@0"
I20260809 07:53:35.354369 2904 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:53:35.356647 2925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:35.357751 2904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:35.357865 2904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "ac5edf032c4e41299ec075f3c4429626"
format_stamp: "Formatted at 2026-08-09 07:53:35 on dist-test-slave-vpvm"
server_key: "4ecd1a9a0cd99c1a245a1673cb8c31b8"
server_key_iv: "9bafabe8ac3e50a6816141def0bb61d9"
server_key_version: "encryptionkey@0"
I20260809 07:53:35.357995 2904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:35.379777 2904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:35.382386 2904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:35.382565 2904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:35.383100 2904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:35.383924 2904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:35.383971 2904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:35.384023 2904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:35.384052 2904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:35.392706 2904 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:36743
I20260809 07:53:35.392760 3095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:36743 every 8 connection(s)
I20260809 07:53:35.393273 2904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:53:35.396812 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 2904
I20260809 07:53:35.396915 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:53:35.397200 29120 external_mini_cluster.cc:1428] Setting key 64e730b026f3b6300e703c59e1a61b92
Aug 09 07:53:35 dist-test-slave-vpvm krb5kdc[2745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:35.406986 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.394960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:49667 (local address 127.28.112.62:43179)
0809 07:53:35.395084 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:35.395088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:35.395919 (+ 831us) server_negotiation.cc:407] Connection header received
0809 07:53:35.397289 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:35.397293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:35.397348 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:35.397440 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:35.398695 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.399398 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.400641 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.400965 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.401072 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:35.403361 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:35.403365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:35.403368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:35.403398 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:35.404963 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:35.405438 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:35.405442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:35.405444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:35.405536 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:35.405827 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:35.405831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:35.405833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:35.406078 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:35.406197 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:35.406684 (+ 487us) server_negotiation.cc:299] Negotiation successful
0809 07:53:35.406813 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.28.112.2@KRBTEST.COM; defaulting to no policy
I20260809 07:53:35.407720 3102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.395271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43179 (local address 127.28.112.1:49667)
0809 07:53:35.395771 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:35.395812 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:35.397078 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:35.397566 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:35.397572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:35.398018 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:35.398553 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:35.398570 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.399530 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.399533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:35.400514 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:35.400527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.401112 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.401115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:35.401254 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:35.401776 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:35.401791 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:35.403199 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:35.405089 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:35.405095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:35.405106 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:35.405344 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:35.405623 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:35.405625 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:35.405626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:35.405742 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:35.406196 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:35.406202 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:35.406434 (+ 232us) client_negotiation.cc:769] Sending connection context
0809 07:53:35.406608 (+ 174us) client_negotiation.cc:240] Negotiation successful
0809 07:53:35.406857 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":120,"threads_started":1}
I20260809 07:53:35.408815 3096 heartbeater.cc:347] Connected to a master server at 127.28.112.62:43179
I20260809 07:53:35.409036 3096 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:35.409494 3096 heartbeater.cc:511] Master 127.28.112.62:43179 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.2@KRBTEST.COM" created.
I20260809 07:53:35.411036 2807 ts_manager.cc:194] Registered new tserver with Master: ac5edf032c4e41299ec075f3c4429626 (127.28.112.1:36743)
I20260809 07:53:35.412523 2807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:49667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
I20260809 07:53:35.424909 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:43179
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:35.532450 3107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:35.532668 3107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:35.532708 3107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:35.536569 3107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:35.536618 3107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:35.536671 3107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:53:35.541723 3107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:43179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:35.542603 3107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:35.543494 3107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:35.552292 3115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:35.552320 3118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:35.552353 3116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:35.552577 3107 server_base.cc:1034] running on GCE node
I20260809 07:53:35.552978 3107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:35.553606 3107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:35.554767 3107 hybrid_clock.cc:648] HybridClock initialized: now 1786262015554751 us; error 28 us; skew 500 ppm
Aug 09 07:53:35 dist-test-slave-vpvm krb5kdc[2745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:35.557433 3107 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:53:35.558188 3107 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:35.558259 3107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:35.558482 3107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:35.560523 3107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "9d9a2b319d4546a88235f155abbcf638"
format_stamp: "Formatted at 2026-08-09 07:53:35 on dist-test-slave-vpvm"
server_key: "e8008d71ebe229640af1005f25b057df"
server_key_iv: "b93c27257853dc25a3d9854a704f958a"
server_key_version: "encryptionkey@0"
I20260809 07:53:35.561110 3107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "9d9a2b319d4546a88235f155abbcf638"
format_stamp: "Formatted at 2026-08-09 07:53:35 on dist-test-slave-vpvm"
server_key: "e8008d71ebe229640af1005f25b057df"
server_key_iv: "b93c27257853dc25a3d9854a704f958a"
server_key_version: "encryptionkey@0"
I20260809 07:53:35.564767 3107 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 07:53:35.567186 3131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:35.568351 3107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:35.568501 3107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "9d9a2b319d4546a88235f155abbcf638"
format_stamp: "Formatted at 2026-08-09 07:53:35 on dist-test-slave-vpvm"
server_key: "e8008d71ebe229640af1005f25b057df"
server_key_iv: "b93c27257853dc25a3d9854a704f958a"
server_key_version: "encryptionkey@0"
I20260809 07:53:35.568602 3107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:35.580030 3107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:35.583232 3107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:35.583413 3107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:35.583997 3107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:35.584915 3107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:35.584965 3107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:35.585016 3107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:35.585045 3107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:35.593261 3107 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:42835
I20260809 07:53:35.593278 3304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:42835 every 8 connection(s)
I20260809 07:53:35.593798 3107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:53:35.600603 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 3107
I20260809 07:53:35.600746 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:53:35.601032 29120 external_mini_cluster.cc:1428] Setting key c22aa75bc1c8034e20db2a750f9a7df5
Aug 09 07:53:35 dist-test-slave-vpvm krb5kdc[2745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:35.606844 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.595393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:56735 (local address 127.28.112.62:43179)
0809 07:53:35.595516 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:35.595521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:35.596345 (+ 824us) server_negotiation.cc:407] Connection header received
0809 07:53:35.597626 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:35.597629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:35.597678 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:35.597748 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:35.598812 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.599329 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.600354 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.600648 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.600760 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:35.603540 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:35.603544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:35.603546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:35.603569 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:35.604966 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:35.605424 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:35.605428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:35.605430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:35.605531 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:35.605818 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:35.605821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:35.605823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:35.606055 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:35.606149 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:35.606585 (+ 436us) server_negotiation.cc:299] Negotiation successful
0809 07:53:35.606697 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 07:53:35.607556 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.595740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43179 (local address 127.28.112.2:56735)
0809 07:53:35.596200 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:35.596245 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:35.597450 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:35.597881 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:35.597887 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:35.598248 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:35.598691 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:35.598704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.599417 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.599419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:35.600263 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:35.600271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.600853 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.600858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:35.601060 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:35.601579 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:35.601595 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:35.603365 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:35.605079 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:35.605084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:35.605095 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:35.605333 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:35.605610 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:35.605612 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:35.605614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:35.605738 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:35.606142 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:35.606148 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:35.606360 (+ 212us) client_negotiation.cc:769] Sending connection context
0809 07:53:35.606510 (+ 150us) client_negotiation.cc:240] Negotiation successful
0809 07:53:35.606749 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":122,"threads_started":1}
I20260809 07:53:35.608561 3306 heartbeater.cc:347] Connected to a master server at 127.28.112.62:43179
I20260809 07:53:35.608791 3306 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:35.609238 3306 heartbeater.cc:511] Master 127.28.112.62:43179 requested a full tablet report, sending...
I20260809 07:53:35.610301 2807 ts_manager.cc:194] Registered new tserver with Master: 9d9a2b319d4546a88235f155abbcf638 (127.28.112.2:42835)
I20260809 07:53:35.610847 2807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:56735
WARNING: no policy specified for kudu/127.28.112.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
I20260809 07:53:35.628391 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:43179
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:35.736552 3313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:35.736773 3313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:35.736814 3313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:35.740684 3313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:35.740733 3313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:35.740787 3313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:53:35.745704 3313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:43179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:35.746603 3313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:35.747516 3313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:35.756388 3319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:35.756389 3324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:35.756388 3322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:35.756743 3313 server_base.cc:1034] running on GCE node
I20260809 07:53:35.757143 3313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:35.757720 3313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:35.758883 3313 hybrid_clock.cc:648] HybridClock initialized: now 1786262015758845 us; error 48 us; skew 500 ppm
Aug 09 07:53:35 dist-test-slave-vpvm krb5kdc[2745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:35.761451 3313 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:53:35.762179 3313 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:35.762248 3313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:35.762449 3313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:35.764338 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "8535a6a19d614e88b75fdf70231bd453"
format_stamp: "Formatted at 2026-08-09 07:53:35 on dist-test-slave-vpvm"
server_key: "447ffce3d472d33d98fad904131aac1b"
server_key_iv: "752fbc95db56193e06bd7f81ae4b2aee"
server_key_version: "encryptionkey@0"
I20260809 07:53:35.764846 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "8535a6a19d614e88b75fdf70231bd453"
format_stamp: "Formatted at 2026-08-09 07:53:35 on dist-test-slave-vpvm"
server_key: "447ffce3d472d33d98fad904131aac1b"
server_key_iv: "752fbc95db56193e06bd7f81ae4b2aee"
server_key_version: "encryptionkey@0"
I20260809 07:53:35.768236 3313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 07:53:35.770476 3330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:35.771575 3313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:35.771678 3313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "8535a6a19d614e88b75fdf70231bd453"
format_stamp: "Formatted at 2026-08-09 07:53:35 on dist-test-slave-vpvm"
server_key: "447ffce3d472d33d98fad904131aac1b"
server_key_iv: "752fbc95db56193e06bd7f81ae4b2aee"
server_key_version: "encryptionkey@0"
I20260809 07:53:35.771780 3313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:35.781912 3313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:35.784294 3313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:35.784454 3313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:35.784936 3313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:35.785676 3313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:35.785722 3313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:35.785769 3313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:35.785796 3313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:35.793928 3313 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:45085
I20260809 07:53:35.793993 3510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:45085 every 8 connection(s)
I20260809 07:53:35.794474 3313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
Aug 09 07:53:35 dist-test-slave-vpvm krb5kdc[2745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:35.804245 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 3313
I20260809 07:53:35.804339 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:53:35.804649 29120 external_mini_cluster.cc:1428] Setting key 6e55d6c9fe58f917b2d0f32e39308631
I20260809 07:53:35.807008 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.796028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:40971 (local address 127.28.112.62:43179)
0809 07:53:35.796182 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:35.796187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:35.797028 (+ 841us) server_negotiation.cc:407] Connection header received
0809 07:53:35.798300 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:35.798303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:35.798351 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:35.798432 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:35.799436 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.799946 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.800946 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.801188 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.801287 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:35.803460 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:35.803463 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:35.803465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:35.803489 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:35.804936 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:35.805457 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:35.805461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:35.805462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:35.805557 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:35.805894 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:35.805898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:35.805899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:35.806127 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:35.806213 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:35.806790 (+ 577us) server_negotiation.cc:299] Negotiation successful
0809 07:53:35.806893 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 07:53:35.807730 3517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.796391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43179 (local address 127.28.112.3:40971)
0809 07:53:35.796875 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:35.796915 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:35.798088 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:35.798560 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:35.798567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:35.798885 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:35.799325 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:35.799338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.800029 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.800031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:35.800857 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:35.800866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.801305 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.801307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:35.801436 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:35.801921 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:35.801969 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:35.803303 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:35.805049 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:35.805055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:35.805067 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:35.805351 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:35.805643 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:35.805646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:35.805647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:35.805797 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:35.806239 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:35.806245 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:35.806531 (+ 286us) client_negotiation.cc:769] Sending connection context
0809 07:53:35.806724 (+ 193us) client_negotiation.cc:240] Negotiation successful
0809 07:53:35.806962 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":119,"threads_started":1}
I20260809 07:53:35.808748 3511 heartbeater.cc:347] Connected to a master server at 127.28.112.62:43179
I20260809 07:53:35.808941 3511 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:35.809324 3511 heartbeater.cc:511] Master 127.28.112.62:43179 requested a full tablet report, sending...
I20260809 07:53:35.810279 2807 ts_manager.cc:194] Registered new tserver with Master: 8535a6a19d614e88b75fdf70231bd453 (127.28.112.3:45085)
I20260809 07:53:35.810815 2807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:40971
I20260809 07:53:35.818184 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 07:53:35.826363 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.819991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39054 (local address 127.28.112.62:43179)
0809 07:53:35.820108 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:35.820111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:35.820256 (+ 145us) server_negotiation.cc:407] Connection header received
0809 07:53:35.820363 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:35.820365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:35.820400 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:35.820477 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:35.821092 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.821633 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.822412 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.822634 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.822713 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:35.823843 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:35.823846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:35.823848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:35.823870 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:35.824938 (+ 1068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:35.825322 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:35.825325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:35.825326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:35.825405 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:35.825617 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:35.825619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:35.825620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:35.825808 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:35.825905 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:35.826145 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 07:53:35.826252 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 07:53:35.828922 2807 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:53:35.831859 2807 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:53:35.845934 3532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.840088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:36743 (local address 127.0.0.1:32772)
0809 07:53:35.840480 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:35.840493 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:35.840580 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:35.841930 (+ 1350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:35.841933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:35.841987 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:35.842094 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:35.842102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.843446 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.843450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:35.844644 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:35.844651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.845436 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.845440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:35.845544 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:35.845580 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 07:53:35.845684 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 07:53:35.845793 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":41,"thread_start_us":94,"threads_started":1}
I20260809 07:53:35.846110 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.839836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:42835 (local address 127.0.0.1:56820)
0809 07:53:35.840212 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:35.840267 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:35.840404 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:35.841291 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:35.841298 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:35.841324 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:35.841467 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:35.841477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.843040 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.843043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:35.844638 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:35.844651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.845668 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.845672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:35.845751 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:35.845773 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 07:53:35.845837 (+ 64us) client_negotiation.cc:240] Negotiation successful
0809 07:53:35.845926 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":154,"thread_start_us":58,"threads_started":1}
I20260809 07:53:35.846204 3530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.840023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56820 (local address 127.28.112.2:42835)
0809 07:53:35.840741 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:35.840756 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:35.840783 (+ 27us) server_negotiation.cc:407] Connection header received
0809 07:53:35.840869 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:35.840875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:35.841104 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:35.841637 (+ 533us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:35.841780 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.842881 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.844762 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.845516 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.845656 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:35.845776 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:35.845890 (+ 114us) server_negotiation.cc:299] Negotiation successful
0809 07:53:35.846037 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":137,"threads_started":1}
I20260809 07:53:35.846202 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.840591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32772 (local address 127.28.112.1:36743)
0809 07:53:35.841481 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:35.841494 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:35.841518 (+ 24us) server_negotiation.cc:407] Connection header received
0809 07:53:35.841594 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:35.841600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:35.841777 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:35.841922 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:35.842250 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.843308 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.844786 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.845314 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.845415 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:35.845561 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:35.845938 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 07:53:35.846051 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":130,"threads_started":1}
I20260809 07:53:35.847064 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.840929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:45085 (local address 127.0.0.1:39398)
0809 07:53:35.841439 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:35.841453 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:35.841541 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:35.842123 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:35.842126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:35.842155 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:35.842241 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:35.842247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.843231 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.843235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:35.844371 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:35.844382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.845789 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.845793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:35.846628 (+ 835us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:35.846653 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 07:53:35.846862 (+ 209us) client_negotiation.cc:240] Negotiation successful
0809 07:53:35.846938 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":725,"thread_start_us":94,"threads_started":1}
I20260809 07:53:35.847064 3537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.841275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.28.112.3:45085)
0809 07:53:35.841672 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:35.841682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:35.841702 (+ 20us) server_negotiation.cc:407] Connection header received
0809 07:53:35.841766 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:35.841770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:35.841926 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:35.842079 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:35.842347 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.843073 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.844619 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.845450 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.845572 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:35.845720 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:35.846790 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0809 07:53:35.846904 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":93,"threads_started":1}
I20260809 07:53:35.849254 2999 tablet_service.cc:1467] Processing CreateTablet for tablet 71948515971d417699b3e15e536ac66d (DEFAULT_TABLE table=test-table [id=4c5ba0f682c2439daa6f44f87698b3ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:35.849262 3197 tablet_service.cc:1467] Processing CreateTablet for tablet 71948515971d417699b3e15e536ac66d (DEFAULT_TABLE table=test-table [id=4c5ba0f682c2439daa6f44f87698b3ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:35.850252 2999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71948515971d417699b3e15e536ac66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:35.849854 3405 tablet_service.cc:1467] Processing CreateTablet for tablet 71948515971d417699b3e15e536ac66d (DEFAULT_TABLE table=test-table [id=4c5ba0f682c2439daa6f44f87698b3ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:35.850359 3197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71948515971d417699b3e15e536ac66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:35.850816 3405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71948515971d417699b3e15e536ac66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:35.856081 3538 tablet_bootstrap.cc:492] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626: Bootstrap starting.
I20260809 07:53:35.857836 3539 tablet_bootstrap.cc:492] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638: Bootstrap starting.
I20260809 07:53:35.858892 3538 tablet_bootstrap.cc:654] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:35.859558 3539 tablet_bootstrap.cc:654] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:35.859972 3538 log.cc:824] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:35.860587 3539 log.cc:824] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:35.861228 3540 tablet_bootstrap.cc:492] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453: Bootstrap starting.
I20260809 07:53:35.861742 3538 tablet_bootstrap.cc:492] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626: No bootstrap required, opened a new log
I20260809 07:53:35.862213 3538 ts_tablet_manager.cc:1397] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 07:53:35.862620 3539 tablet_bootstrap.cc:492] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638: No bootstrap required, opened a new log
I20260809 07:53:35.862849 3539 ts_tablet_manager.cc:1397] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 07:53:35.863168 3540 tablet_bootstrap.cc:654] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:35.864157 3540 log.cc:824] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:35.865715 3540 tablet_bootstrap.cc:492] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453: No bootstrap required, opened a new log
I20260809 07:53:35.865888 3540 ts_tablet_manager.cc:1397] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260809 07:53:35.866292 3538 raft_consensus.cc:348] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9a2b319d4546a88235f155abbcf638" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 42835 } } peers { permanent_uuid: "ac5edf032c4e41299ec075f3c4429626" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36743 } } peers { permanent_uuid: "8535a6a19d614e88b75fdf70231bd453" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 45085 } }
I20260809 07:53:35.866567 3538 raft_consensus.cc:374] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:35.866415 3539 raft_consensus.cc:348] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9a2b319d4546a88235f155abbcf638" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 42835 } } peers { permanent_uuid: "ac5edf032c4e41299ec075f3c4429626" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36743 } } peers { permanent_uuid: "8535a6a19d614e88b75fdf70231bd453" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 45085 } }
I20260809 07:53:35.866645 3539 raft_consensus.cc:374] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:35.866667 3538 raft_consensus.cc:729] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac5edf032c4e41299ec075f3c4429626, State: Initialized, Role: FOLLOWER
I20260809 07:53:35.866703 3539 raft_consensus.cc:729] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d9a2b319d4546a88235f155abbcf638, State: Initialized, Role: FOLLOWER
I20260809 07:53:35.867130 3539 consensus_queue.cc:260] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9a2b319d4546a88235f155abbcf638" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 42835 } } peers { permanent_uuid: "ac5edf032c4e41299ec075f3c4429626" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36743 } } peers { permanent_uuid: "8535a6a19d614e88b75fdf70231bd453" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 45085 } }
I20260809 07:53:35.867130 3538 consensus_queue.cc:260] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9a2b319d4546a88235f155abbcf638" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 42835 } } peers { permanent_uuid: "ac5edf032c4e41299ec075f3c4429626" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36743 } } peers { permanent_uuid: "8535a6a19d614e88b75fdf70231bd453" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 45085 } }
I20260809 07:53:35.868012 3306 heartbeater.cc:503] Master 127.28.112.62:43179 was elected leader, sending a full tablet report...
I20260809 07:53:35.868083 3096 heartbeater.cc:503] Master 127.28.112.62:43179 was elected leader, sending a full tablet report...
I20260809 07:53:35.868106 3539 ts_tablet_manager.cc:1428] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 07:53:35.869307 3548 raft_consensus.cc:482] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:35.869504 3548 raft_consensus.cc:504] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9a2b319d4546a88235f155abbcf638" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 42835 } } peers { permanent_uuid: "ac5edf032c4e41299ec075f3c4429626" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36743 } } peers { permanent_uuid: "8535a6a19d614e88b75fdf70231bd453" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 45085 } }
I20260809 07:53:35.869576 3540 raft_consensus.cc:348] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9a2b319d4546a88235f155abbcf638" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 42835 } } peers { permanent_uuid: "ac5edf032c4e41299ec075f3c4429626" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36743 } } peers { permanent_uuid: "8535a6a19d614e88b75fdf70231bd453" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 45085 } }
I20260809 07:53:35.869835 3540 raft_consensus.cc:374] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:35.869908 3540 raft_consensus.cc:729] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8535a6a19d614e88b75fdf70231bd453, State: Initialized, Role: FOLLOWER
I20260809 07:53:35.870450 3540 consensus_queue.cc:260] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9a2b319d4546a88235f155abbcf638" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 42835 } } peers { permanent_uuid: "ac5edf032c4e41299ec075f3c4429626" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36743 } } peers { permanent_uuid: "8535a6a19d614e88b75fdf70231bd453" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 45085 } }
I20260809 07:53:35.871443 3540 ts_tablet_manager.cc:1428] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 07:53:35.871800 3511 heartbeater.cc:503] Master 127.28.112.62:43179 was elected leader, sending a full tablet report...
I20260809 07:53:35.872901 3548 leader_election.cc:288] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d9a2b319d4546a88235f155abbcf638 (127.28.112.2:42835), 8535a6a19d614e88b75fdf70231bd453 (127.28.112.3:45085)
I20260809 07:53:35.874073 3538 ts_tablet_manager.cc:1428] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260809 07:53:35.875725 3102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.871004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:45085 (local address 127.28.112.1:56897)
0809 07:53:35.871142 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:35.871158 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:35.871267 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:35.871948 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:35.871951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:35.871970 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:35.872089 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:35.872097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.873171 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.873175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:35.874317 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:35.874329 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.875278 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.875282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:35.875431 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:35.875455 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 07:53:35.875526 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 07:53:35.875597 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":18}
I20260809 07:53:35.876235 3537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.871279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:56897 (local address 127.28.112.3:45085)
0809 07:53:35.871390 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:35.871395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:35.871411 (+ 16us) server_negotiation.cc:407] Connection header received
0809 07:53:35.871461 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:35.871464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:35.871515 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:35.871598 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:35.872336 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.873032 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.874513 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.875111 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.875237 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:35.875296 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:35.876024 (+ 728us) server_negotiation.cc:299] Negotiation successful
0809 07:53:35.876099 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 07:53:35.876735 3435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71948515971d417699b3e15e536ac66d" candidate_uuid: "ac5edf032c4e41299ec075f3c4429626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8535a6a19d614e88b75fdf70231bd453" is_pre_election: true
I20260809 07:53:35.877024 3435 raft_consensus.cc:2454] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac5edf032c4e41299ec075f3c4429626 in term 0.
I20260809 07:53:35.877170 3553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.871587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:42835 (local address 127.28.112.1:39583)
0809 07:53:35.873160 (+ 1573us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:35.873174 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:35.873268 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:35.873744 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:35.873748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:35.873766 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:35.873891 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:35.873897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.874838 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.874840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:35.875980 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:35.875991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.876829 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.876832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:35.876901 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:35.876921 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 07:53:35.876979 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 07:53:35.877026 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1507,"mutex_wait_us":19,"thread_start_us":94,"threads_started":1}
I20260809 07:53:35.877202 3530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.871946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:39583 (local address 127.28.112.2:42835)
0809 07:53:35.872276 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:35.872281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:35.873520 (+ 1239us) server_negotiation.cc:407] Connection header received
0809 07:53:35.873576 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:35.873579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:35.873626 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:35.873710 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:35.874067 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.874706 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.876103 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.876693 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.876830 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:35.876884 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:35.877015 (+ 131us) server_negotiation.cc:299] Negotiation successful
0809 07:53:35.877089 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260809 07:53:35.877651 2930 leader_election.cc:302] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8535a6a19d614e88b75fdf70231bd453, ac5edf032c4e41299ec075f3c4429626; no voters:
I20260809 07:53:35.877732 3225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71948515971d417699b3e15e536ac66d" candidate_uuid: "ac5edf032c4e41299ec075f3c4429626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d9a2b319d4546a88235f155abbcf638" is_pre_election: true
I20260809 07:53:35.877978 3548 raft_consensus.cc:2781] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:53:35.878000 3225 raft_consensus.cc:2454] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac5edf032c4e41299ec075f3c4429626 in term 0.
I20260809 07:53:35.878072 3548 raft_consensus.cc:482] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:53:35.878132 3548 raft_consensus.cc:3037] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:35.879123 3548 raft_consensus.cc:504] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9a2b319d4546a88235f155abbcf638" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 42835 } } peers { permanent_uuid: "ac5edf032c4e41299ec075f3c4429626" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36743 } } peers { permanent_uuid: "8535a6a19d614e88b75fdf70231bd453" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 45085 } }
I20260809 07:53:35.879448 3548 leader_election.cc:288] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [CANDIDATE]: Term 1 election: Requested vote from peers 9d9a2b319d4546a88235f155abbcf638 (127.28.112.2:42835), 8535a6a19d614e88b75fdf70231bd453 (127.28.112.3:45085)
I20260809 07:53:35.879767 3225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71948515971d417699b3e15e536ac66d" candidate_uuid: "ac5edf032c4e41299ec075f3c4429626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d9a2b319d4546a88235f155abbcf638"
I20260809 07:53:35.879766 3435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71948515971d417699b3e15e536ac66d" candidate_uuid: "ac5edf032c4e41299ec075f3c4429626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8535a6a19d614e88b75fdf70231bd453"
I20260809 07:53:35.879889 3435 raft_consensus.cc:3037] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:35.879889 3225 raft_consensus.cc:3037] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:35.881122 3225 raft_consensus.cc:2454] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac5edf032c4e41299ec075f3c4429626 in term 1.
I20260809 07:53:35.881122 3435 raft_consensus.cc:2454] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac5edf032c4e41299ec075f3c4429626 in term 1.
I20260809 07:53:35.881444 2929 leader_election.cc:302] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8535a6a19d614e88b75fdf70231bd453, 9d9a2b319d4546a88235f155abbcf638, ac5edf032c4e41299ec075f3c4429626; no voters:
I20260809 07:53:35.881623 3548 raft_consensus.cc:2781] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:35.881721 3548 raft_consensus.cc:686] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [term 1 LEADER]: Becoming Leader. State: Replica: ac5edf032c4e41299ec075f3c4429626, State: Running, Role: LEADER
I20260809 07:53:35.882030 3548 consensus_queue.cc:237] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9a2b319d4546a88235f155abbcf638" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 42835 } } peers { permanent_uuid: "ac5edf032c4e41299ec075f3c4429626" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36743 } } peers { permanent_uuid: "8535a6a19d614e88b75fdf70231bd453" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 45085 } }
I20260809 07:53:35.884742 2804 catalog_manager.cc:5495] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac5edf032c4e41299ec075f3c4429626 (127.28.112.1). New cstate: current_term: 1 leader_uuid: "ac5edf032c4e41299ec075f3c4429626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d9a2b319d4546a88235f155abbcf638" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 42835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac5edf032c4e41299ec075f3c4429626" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8535a6a19d614e88b75fdf70231bd453" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 45085 } health_report { overall_health: UNKNOWN } } }
W20260809 07:53:35.894254 3099 tablet.cc:2369] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:53:35.895145 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.892090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32776 (local address 127.28.112.1:36743)
0809 07:53:35.892187 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:35.892190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:35.892203 (+ 13us) server_negotiation.cc:407] Connection header received
0809 07:53:35.892324 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:35.892326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:35.892371 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:35.892469 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:35.892653 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.893109 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.893857 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:35.894118 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:35.894215 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:35.894447 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:35.894870 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:35.894960 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:35.895011 (+ 51us) server_negotiation.cc:299] Negotiation successful
0809 07:53:35.895055 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260809 07:53:35.900141 3435 raft_consensus.cc:1261] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453 [term 1 FOLLOWER]: Refusing update from remote peer ac5edf032c4e41299ec075f3c4429626: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:35.900141 3225 raft_consensus.cc:1261] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638 [term 1 FOLLOWER]: Refusing update from remote peer ac5edf032c4e41299ec075f3c4429626: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:35.900779 3551 consensus_queue.cc:1035] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8535a6a19d614e88b75fdf70231bd453" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 45085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:35.900940 3548 consensus_queue.cc:1035] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d9a2b319d4546a88235f155abbcf638" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 42835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:35.916352 2805 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:35.916535 2805 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:35.918946 2805 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d732235db464b01925f0490534bd29a: Sending DeleteTablet for 3 replicas of tablet 71948515971d417699b3e15e536ac66d
I20260809 07:53:35.919672 3197 tablet_service.cc:1514] Processing DeleteTablet for tablet 71948515971d417699b3e15e536ac66d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:35 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:56820
I20260809 07:53:35.919807 2999 tablet_service.cc:1514] Processing DeleteTablet for tablet 71948515971d417699b3e15e536ac66d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:35 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:32772
I20260809 07:53:35.920177 3567 tablet_replica.cc:330] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638: stopping tablet replica
I20260809 07:53:35.920243 3569 tablet_replica.cc:330] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626: stopping tablet replica
I20260809 07:53:35.920190 3405 tablet_service.cc:1514] Processing DeleteTablet for tablet 71948515971d417699b3e15e536ac66d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:35 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:39398
I20260809 07:53:35.920528 3567 raft_consensus.cc:2229] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:35.920528 3569 raft_consensus.cc:2229] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:53:35.920879 3569 raft_consensus.cc:2258] T 71948515971d417699b3e15e536ac66d P ac5edf032c4e41299ec075f3c4429626 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:35.920886 3567 raft_consensus.cc:2258] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:35.921281 3572 tablet_replica.cc:330] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453: stopping tablet replica
I20260809 07:53:35.921396 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 2904
I20260809 07:53:35.921566 3572 raft_consensus.cc:2229] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:35.921789 3572 raft_consensus.cc:2258] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:35.928005 3572 ts_tablet_manager.cc:1905] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:35.931043 3572 ts_tablet_manager.cc:1918] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:53:35.931147 3572 log.cc:1196] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/71948515971d417699b3e15e536ac66d
I20260809 07:53:35.931418 3572 ts_tablet_manager.cc:1939] T 71948515971d417699b3e15e536ac66d P 8535a6a19d614e88b75fdf70231bd453: Deleting consensus metadata
I20260809 07:53:35.932966 2788 catalog_manager.cc:4841] TS 8535a6a19d614e88b75fdf70231bd453 (127.28.112.3:45085): tablet 71948515971d417699b3e15e536ac66d (table test-table [id=4c5ba0f682c2439daa6f44f87698b3ce]) successfully deleted
I20260809 07:53:35.934779 3567 ts_tablet_manager.cc:1905] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:35.937074 3567 ts_tablet_manager.cc:1918] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:53:35.937157 3567 log.cc:1196] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/71948515971d417699b3e15e536ac66d
I20260809 07:53:35.937461 3567 ts_tablet_manager.cc:1939] T 71948515971d417699b3e15e536ac66d P 9d9a2b319d4546a88235f155abbcf638: Deleting consensus metadata
I20260809 07:53:35.938621 2786 catalog_manager.cc:4841] TS 9d9a2b319d4546a88235f155abbcf638 (127.28.112.2:42835): tablet 71948515971d417699b3e15e536ac66d (table test-table [id=4c5ba0f682c2439daa6f44f87698b3ce]) successfully deleted
W20260809 07:53:35.940788 2788 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.112.1:36743 (error 108)
I20260809 07:53:35.940863 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 3107
I20260809 07:53:35.942080 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:35.941753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:36743 (local address 127.0.0.1:32782)
0809 07:53:35.941913 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:35.942003 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:36743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260809 07:53:35.942224 2788 catalog_manager.cc:4584] TS ac5edf032c4e41299ec075f3c4429626 (127.28.112.1:36743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71948515971d417699b3e15e536ac66d: Network error: Client connection negotiation failed: client connection to 127.28.112.1:36743: connect: Connection refused (error 111)
I20260809 07:53:35.952836 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 3313
I20260809 07:53:35.963073 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 2763
2026-08-09T07:53:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2266 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-09T07:53:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:53:35Z Disabled control of system clock
I20260809 07:53:35.996845 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:40329
--webserver_interface=127.28.112.62
--webserver_port=0
--builtin_ntp_servers=127.28.112.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:40329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 07:53:36.107331 3577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:36.107592 3577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:36.107630 3577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:36.111519 3577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:36.111573 3577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:36.111590 3577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:36.111609 3577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:36.111629 3577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:53:36.116184 3577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:40329
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:40329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:36.117102 3577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:36.118422 3577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:36.126475 3590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:36.126524 3577 server_base.cc:1034] running on GCE node
W20260809 07:53:36.126477 3587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:36.126484 3588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:36.127224 3577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:36.128214 3577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:36.129374 3577 hybrid_clock.cc:648] HybridClock initialized: now 1786262016129358 us; error 27 us; skew 500 ppm
I20260809 07:53:36.131143 3577 webserver.cc:460] Webserver started at http://127.28.112.62:45407/ using document root <none> and password file <none>
I20260809 07:53:36.131704 3577 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:36.131769 3577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:36.131975 3577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:36.133914 3577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "e66a91921ab744db81708c6a2991b1a9"
format_stamp: "Formatted at 2026-08-09 07:53:36 on dist-test-slave-vpvm"
server_key: "cb2b1ad25efc47c28344e7f1acec66ad"
server_key_iv: "f52f8d7c9945ab5c9ff5cdb6236efb67"
server_key_version: "encryptionkey@0"
I20260809 07:53:36.134490 3577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "e66a91921ab744db81708c6a2991b1a9"
format_stamp: "Formatted at 2026-08-09 07:53:36 on dist-test-slave-vpvm"
server_key: "cb2b1ad25efc47c28344e7f1acec66ad"
server_key_iv: "f52f8d7c9945ab5c9ff5cdb6236efb67"
server_key_version: "encryptionkey@0"
I20260809 07:53:36.138186 3577 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 07:53:36.140362 3597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:36.141474 3577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:36.141589 3577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "e66a91921ab744db81708c6a2991b1a9"
format_stamp: "Formatted at 2026-08-09 07:53:36 on dist-test-slave-vpvm"
server_key: "cb2b1ad25efc47c28344e7f1acec66ad"
server_key_iv: "f52f8d7c9945ab5c9ff5cdb6236efb67"
server_key_version: "encryptionkey@0"
I20260809 07:53:36.141683 3577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:36.145931 3577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:36.146507 3577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:36.146653 3577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:36.152750 3577 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:40329
I20260809 07:53:36.152756 3682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:40329 every 8 connection(s)
I20260809 07:53:36.153565 3577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:53:36.156629 3684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:36.161753 3684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9: Bootstrap starting.
I20260809 07:53:36.162214 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 3577
I20260809 07:53:36.162333 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:53:36.162597 29120 external_mini_cluster.cc:1428] Setting key e10130f874d66de8a96ecddb86c64c87
I20260809 07:53:36.164422 3684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:36.165095 3684 log.cc:824] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:36.167193 3684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9: No bootstrap required, opened a new log
I20260809 07:53:36.167821 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:36.163859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.28.112.62:40329)
0809 07:53:36.164462 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:36.164476 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:36.164511 (+ 35us) server_negotiation.cc:407] Connection header received
0809 07:53:36.165121 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:36.165137 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:36.165510 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:36.165846 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:36.166639 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:36.166644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:36.166654 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:53:36.166665 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:36.166716 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:53:36.166730 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:36.166807 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:36.166937 (+ 130us) server_negotiation.cc:299] Negotiation successful
0809 07:53:36.167152 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":126,"threads_started":1}
I20260809 07:53:36.170876 3684 raft_consensus.cc:348] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66a91921ab744db81708c6a2991b1a9" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40329 } }
I20260809 07:53:36.171074 3684 raft_consensus.cc:374] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:36.171130 3684 raft_consensus.cc:729] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e66a91921ab744db81708c6a2991b1a9, State: Initialized, Role: FOLLOWER
I20260809 07:53:36.171602 3684 consensus_queue.cc:260] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66a91921ab744db81708c6a2991b1a9" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40329 } }
I20260809 07:53:36.171753 3684 raft_consensus.cc:388] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:53:36.171818 3684 raft_consensus.cc:482] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:53:36.171914 3684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:36.172700 3684 raft_consensus.cc:504] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66a91921ab744db81708c6a2991b1a9" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40329 } }
I20260809 07:53:36.173005 3684 leader_election.cc:302] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e66a91921ab744db81708c6a2991b1a9; no voters:
I20260809 07:53:36.173285 3684 leader_election.cc:288] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:53:36.173362 3697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:36.173581 3697 raft_consensus.cc:686] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9 [term 1 LEADER]: Becoming Leader. State: Replica: e66a91921ab744db81708c6a2991b1a9, State: Running, Role: LEADER
I20260809 07:53:36.173929 3697 consensus_queue.cc:237] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66a91921ab744db81708c6a2991b1a9" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40329 } }
I20260809 07:53:36.174537 3684 sys_catalog.cc:564] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:36.175704 3700 sys_catalog.cc:455] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e66a91921ab744db81708c6a2991b1a9. Latest consensus state: current_term: 1 leader_uuid: "e66a91921ab744db81708c6a2991b1a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66a91921ab744db81708c6a2991b1a9" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40329 } } }
I20260809 07:53:36.175737 3698 sys_catalog.cc:455] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e66a91921ab744db81708c6a2991b1a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66a91921ab744db81708c6a2991b1a9" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40329 } } }
I20260809 07:53:36.175933 3698 sys_catalog.cc:458] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:36.175915 3700 sys_catalog.cc:458] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:36.177156 3708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:53:36.179832 3708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:36.185060 3708 catalog_manager.cc:1347] Generated new cluster ID: 5f7ee70072824d93a388e0262a167923
I20260809 07:53:36.185113 3708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:36.195178 3708 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:53:36.195430 3708 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:36.199855 3708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9: Generated new TSK 0
I20260809 07:53:36.200549 3708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:53:36.214205 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:40329
--builtin_ntp_servers=127.28.112.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 07:53:36.357568 3722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:36.357815 3722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:36.357860 3722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:36.361804 3722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:36.361860 3722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:36.361970 3722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:53:36.366619 3722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:40329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:36.367636 3722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:36.369019 3722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:36.378306 3731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:36.378299 3728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:36.378387 3722 server_base.cc:1034] running on GCE node
W20260809 07:53:36.378299 3734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:36.378904 3722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:36.379482 3722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:36.380657 3722 hybrid_clock.cc:648] HybridClock initialized: now 1786262016380638 us; error 29 us; skew 500 ppm
I20260809 07:53:36.382491 3722 webserver.cc:460] Webserver started at http://127.28.112.1:33009/ using document root <none> and password file <none>
I20260809 07:53:36.383028 3722 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:36.383098 3722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:36.383320 3722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:36.385279 3722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "74f6dfcaaa994aefa23a7429fd77397e"
format_stamp: "Formatted at 2026-08-09 07:53:36 on dist-test-slave-vpvm"
server_key: "c6deb47c64de78864259f1adf9443f2b"
server_key_iv: "d16a4f859177d5fb41ceee0728f227e6"
server_key_version: "encryptionkey@0"
I20260809 07:53:36.385823 3722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "74f6dfcaaa994aefa23a7429fd77397e"
format_stamp: "Formatted at 2026-08-09 07:53:36 on dist-test-slave-vpvm"
server_key: "c6deb47c64de78864259f1adf9443f2b"
server_key_iv: "d16a4f859177d5fb41ceee0728f227e6"
server_key_version: "encryptionkey@0"
I20260809 07:53:36.389503 3722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 07:53:36.391875 3743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:36.393028 3722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:36.393133 3722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "74f6dfcaaa994aefa23a7429fd77397e"
format_stamp: "Formatted at 2026-08-09 07:53:36 on dist-test-slave-vpvm"
server_key: "c6deb47c64de78864259f1adf9443f2b"
server_key_iv: "d16a4f859177d5fb41ceee0728f227e6"
server_key_version: "encryptionkey@0"
I20260809 07:53:36.393224 3722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:36.397744 3722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:36.398360 3722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:36.398533 3722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:36.399057 3722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:36.399946 3722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:36.399998 3722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:36.400053 3722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:36.400085 3722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:36.408876 3722 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:40097
I20260809 07:53:36.408897 3922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:40097 every 8 connection(s)
I20260809 07:53:36.409735 3722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:53:36.412827 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 3722
I20260809 07:53:36.412953 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:53:36.413252 29120 external_mini_cluster.cc:1428] Setting key ecf49e564ef452ac6873db87d36e1501
I20260809 07:53:36.415395 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:40329
--builtin_ntp_servers=127.28.112.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 07:53:36.416128 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:36.411446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:56491 (local address 127.28.112.62:40329)
0809 07:53:36.411625 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:36.411631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:36.412655 (+ 1024us) server_negotiation.cc:407] Connection header received
0809 07:53:36.414347 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:36.414351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:36.414389 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:36.414512 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:36.415718 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:36.415722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:36.415724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:53:36.415739 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:36.415762 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:53:36.415771 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:36.415852 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:36.415988 (+ 136us) server_negotiation.cc:299] Negotiation successful
0809 07:53:36.416025 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 07:53:36.417155 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:36.411852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:40329 (local address 127.28.112.1:56491)
0809 07:53:36.412504 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:36.412542 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:36.414147 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:36.414699 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:36.414706 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:36.414803 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:36.415363 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:53:36.415365 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:36.415418 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:53:36.415507 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:36.415853 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:36.415861 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:36.415867 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 07:53:36.416032 (+ 165us) client_negotiation.cc:240] Negotiation successful
0809 07:53:36.416261 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":124,"threads_started":1}
I20260809 07:53:36.418380 3923 heartbeater.cc:347] Connected to a master server at 127.28.112.62:40329
I20260809 07:53:36.418591 3923 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:36.418968 3923 heartbeater.cc:511] Master 127.28.112.62:40329 requested a full tablet report, sending...
I20260809 07:53:36.420598 3621 ts_manager.cc:194] Registered new tserver with Master: 74f6dfcaaa994aefa23a7429fd77397e (127.28.112.1:40097)
W20260809 07:53:36.523403 3926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:36.523635 3926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:36.523672 3926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:36.527506 3926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:36.527554 3926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:36.527604 3926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:53:36.532121 3926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:40329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:36.533049 3926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:36.534339 3926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:36.543391 3939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:36.543439 3926 server_base.cc:1034] running on GCE node
W20260809 07:53:36.543401 3935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:36.543413 3934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:36.544071 3926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:36.544721 3926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:36.545876 3926 hybrid_clock.cc:648] HybridClock initialized: now 1786262016545845 us; error 44 us; skew 500 ppm
I20260809 07:53:36.547586 3926 webserver.cc:460] Webserver started at http://127.28.112.2:45715/ using document root <none> and password file <none>
I20260809 07:53:36.548141 3926 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:36.548208 3926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:36.548413 3926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:36.550419 3926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "c180abd9f3bc4f37bd3287c64917b2cc"
format_stamp: "Formatted at 2026-08-09 07:53:36 on dist-test-slave-vpvm"
server_key: "4e83449961a4ae376b08d8c3c89f5b70"
server_key_iv: "86637268b76a3f02f5daa94a30eb7c1f"
server_key_version: "encryptionkey@0"
I20260809 07:53:36.550943 3926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "c180abd9f3bc4f37bd3287c64917b2cc"
format_stamp: "Formatted at 2026-08-09 07:53:36 on dist-test-slave-vpvm"
server_key: "4e83449961a4ae376b08d8c3c89f5b70"
server_key_iv: "86637268b76a3f02f5daa94a30eb7c1f"
server_key_version: "encryptionkey@0"
I20260809 07:53:36.554476 3926 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:53:36.556875 3947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:36.558074 3926 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:36.558182 3926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "c180abd9f3bc4f37bd3287c64917b2cc"
format_stamp: "Formatted at 2026-08-09 07:53:36 on dist-test-slave-vpvm"
server_key: "4e83449961a4ae376b08d8c3c89f5b70"
server_key_iv: "86637268b76a3f02f5daa94a30eb7c1f"
server_key_version: "encryptionkey@0"
I20260809 07:53:36.558282 3926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:36.562780 3926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:36.563385 3926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:36.563546 3926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:36.564073 3926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:36.564973 3926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:36.565021 3926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:36.565073 3926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:36.565102 3926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:36.573442 3926 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:41715
I20260809 07:53:36.573478 4122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:41715 every 8 connection(s)
I20260809 07:53:36.574362 3926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:53:36.580034 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:36.576217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:35335 (local address 127.28.112.62:40329)
0809 07:53:36.576350 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:36.576355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:36.577401 (+ 1046us) server_negotiation.cc:407] Connection header received
0809 07:53:36.578681 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:36.578684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:36.578722 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:36.578790 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:36.579702 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:36.579705 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:36.579707 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:53:36.579720 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:36.579740 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:53:36.579748 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:36.579817 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:36.579895 (+ 78us) server_negotiation.cc:299] Negotiation successful
0809 07:53:36.579930 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 07:53:36.580837 4125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:36.576625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:40329 (local address 127.28.112.2:35335)
0809 07:53:36.577260 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:36.577299 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:36.578499 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:36.578919 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:36.578925 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:36.579008 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:36.579468 (+ 460us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:53:36.579470 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:36.579508 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:53:36.579573 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:36.579802 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:36.579806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:36.579809 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 07:53:36.579924 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 07:53:36.580108 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":112,"threads_started":1}
I20260809 07:53:36.581904 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 3926
I20260809 07:53:36.581911 4123 heartbeater.cc:347] Connected to a master server at 127.28.112.62:40329
I20260809 07:53:36.582053 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:53:36.582149 4123 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:36.582336 29120 external_mini_cluster.cc:1428] Setting key 64a96eb34b8e841d4122f2e9e2b5715a
I20260809 07:53:36.582547 4123 heartbeater.cc:511] Master 127.28.112.62:40329 requested a full tablet report, sending...
I20260809 07:53:36.583603 3620 ts_manager.cc:194] Registered new tserver with Master: c180abd9f3bc4f37bd3287c64917b2cc (127.28.112.2:41715)
I20260809 07:53:36.584296 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:40329
--builtin_ntp_servers=127.28.112.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 07:53:36.693166 4126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:36.693404 4126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:36.693457 4126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:36.697405 4126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:36.697460 4126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:36.697527 4126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:53:36.702144 4126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:40329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:36.703083 4126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:36.704365 4126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:36.713299 4140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:36.713394 4126 server_base.cc:1034] running on GCE node
W20260809 07:53:36.713295 4134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:36.713586 4137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:36.713996 4126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:36.714572 4126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:36.715731 4126 hybrid_clock.cc:648] HybridClock initialized: now 1786262016715716 us; error 30 us; skew 500 ppm
I20260809 07:53:36.717396 4126 webserver.cc:460] Webserver started at http://127.28.112.3:40061/ using document root <none> and password file <none>
I20260809 07:53:36.717937 4126 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:36.718047 4126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:36.718225 4126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:36.720165 4126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "8291fe35f1a14394a3d1101bac5a3c34"
format_stamp: "Formatted at 2026-08-09 07:53:36 on dist-test-slave-vpvm"
server_key: "e5f2ac0c30e351beb16cf535f81dd3e4"
server_key_iv: "b5cb3affb9591233024da40861326421"
server_key_version: "encryptionkey@0"
I20260809 07:53:36.720660 4126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "8291fe35f1a14394a3d1101bac5a3c34"
format_stamp: "Formatted at 2026-08-09 07:53:36 on dist-test-slave-vpvm"
server_key: "e5f2ac0c30e351beb16cf535f81dd3e4"
server_key_iv: "b5cb3affb9591233024da40861326421"
server_key_version: "encryptionkey@0"
I20260809 07:53:36.724084 4126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 07:53:36.726378 4151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:36.727478 4126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:36.727587 4126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "8291fe35f1a14394a3d1101bac5a3c34"
format_stamp: "Formatted at 2026-08-09 07:53:36 on dist-test-slave-vpvm"
server_key: "e5f2ac0c30e351beb16cf535f81dd3e4"
server_key_iv: "b5cb3affb9591233024da40861326421"
server_key_version: "encryptionkey@0"
I20260809 07:53:36.727681 4126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:36.731853 4126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:36.732383 4126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:36.732511 4126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:36.732973 4126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:36.733795 4126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:36.733840 4126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:36.733876 4126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:36.733896 4126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:36.741832 4126 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:35181
I20260809 07:53:36.741854 4335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:35181 every 8 connection(s)
I20260809 07:53:36.742647 4126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:53:36.748219 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:36.744304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:37001 (local address 127.28.112.62:40329)
0809 07:53:36.744449 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:36.744452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:36.745450 (+ 998us) server_negotiation.cc:407] Connection header received
0809 07:53:36.746744 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:36.746748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:36.746793 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:36.746865 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:36.747800 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:36.747803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:36.747806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:53:36.747818 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:36.747838 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:53:36.747846 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:36.747928 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:36.748050 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 07:53:36.748092 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 07:53:36.748942 4340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:36.744670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:40329 (local address 127.28.112.3:37001)
0809 07:53:36.745317 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:36.745353 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:36.746547 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:36.747024 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:36.747031 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:36.747111 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:36.747557 (+ 446us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:53:36.747559 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:36.747595 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:53:36.747656 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:36.747915 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:36.747924 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:36.747930 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 07:53:36.748079 (+ 149us) client_negotiation.cc:240] Negotiation successful
0809 07:53:36.748296 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":127,"threads_started":1}
I20260809 07:53:36.749970 4336 heartbeater.cc:347] Connected to a master server at 127.28.112.62:40329
I20260809 07:53:36.750159 4336 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:36.750528 4336 heartbeater.cc:511] Master 127.28.112.62:40329 requested a full tablet report, sending...
I20260809 07:53:36.750671 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 4126
I20260809 07:53:36.750774 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:53:36.751036 29120 external_mini_cluster.cc:1428] Setting key cfd886261ac97b949b46df1fd237f9ce
I20260809 07:53:36.751585 3620 ts_manager.cc:194] Registered new tserver with Master: 8291fe35f1a14394a3d1101bac5a3c34 (127.28.112.3:35181)
I20260809 07:53:36.752971 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 07:53:36.755949 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:36.754743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.28.112.62:40329)
0809 07:53:36.754839 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:36.754842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:36.755012 (+ 170us) server_negotiation.cc:407] Connection header received
0809 07:53:36.755095 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:36.755096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:36.755122 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:36.755178 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:36.755723 (+ 545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:36.755725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:36.755727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:53:36.755733 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:36.755744 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:53:36.755750 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:36.755802 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:36.755842 (+ 40us) server_negotiation.cc:299] Negotiation successful
0809 07:53:36.755868 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 07:53:36.758472 3620 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:53:36.761181 3620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:53:36.771239 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:36.768653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:40097 (local address 127.0.0.1:44116)
0809 07:53:36.769464 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:36.769510 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:36.769598 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:36.770048 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:36.770054 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:36.770123 (+ 69us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:36.770591 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:53:36.770593 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:36.770611 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:53:36.770664 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:36.770937 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:36.770946 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:36.770952 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 07:53:36.771054 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 07:53:36.771119 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":72,"threads_started":1}
I20260809 07:53:36.771461 4361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:36.768804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.28.112.1:40097)
0809 07:53:36.769620 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:36.769632 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:36.769657 (+ 25us) server_negotiation.cc:407] Connection header received
0809 07:53:36.769708 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:36.769713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:36.769896 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:36.770052 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:36.770775 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:36.770781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:36.770793 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:53:36.770803 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:36.770839 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:53:36.770858 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:36.770947 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:36.771241 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 07:53:36.771314 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":116,"threads_started":1}
I20260809 07:53:36.772177 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:36.769078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:35181 (local address 127.0.0.1:58942)
0809 07:53:36.769971 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:36.769984 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:36.770073 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:36.771071 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:36.771074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:36.771090 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:36.771527 (+ 437us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:53:36.771529 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:36.771539 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:53:36.771566 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:36.771958 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:36.771961 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:36.771964 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 07:53:36.772030 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 07:53:36.772065 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"spinlock_wait_cycles":6656,"thread_start_us":92,"threads_started":1}
I20260809 07:53:36.772595 4365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:36.769387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34506 (local address 127.28.112.2:41715)
0809 07:53:36.769811 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:36.769824 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:36.770165 (+ 341us) server_negotiation.cc:407] Connection header received
0809 07:53:36.770220 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:36.770225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:36.770671 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:36.770802 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:36.771890 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:36.771894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:36.771901 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:53:36.771908 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:36.771939 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:53:36.771953 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:36.772049 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:36.772390 (+ 341us) server_negotiation.cc:299] Negotiation successful
0809 07:53:36.772457 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":111,"threads_started":1}
I20260809 07:53:36.772608 4363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:36.769217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58942 (local address 127.28.112.3:35181)
0809 07:53:36.769799 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:36.769812 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:36.770172 (+ 360us) server_negotiation.cc:407] Connection header received
0809 07:53:36.770219 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:36.770225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:36.770391 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:36.770508 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:36.771695 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:36.771701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:36.771711 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:53:36.771722 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:36.771759 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:53:36.771785 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:36.771887 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:36.772414 (+ 527us) server_negotiation.cc:299] Negotiation successful
0809 07:53:36.772476 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":78,"threads_started":1}
I20260809 07:53:36.772871 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:36.768811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:41715 (local address 127.0.0.1:34506)
0809 07:53:36.769946 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:36.769985 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:36.770073 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:36.771012 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:36.771015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:36.771030 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:36.771534 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:53:36.771536 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:36.771548 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:53:36.771580 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:36.772096 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:36.772100 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:36.772103 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 07:53:36.772175 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 07:53:36.772211 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"spinlock_wait_cycles":4096,"thread_start_us":96,"threads_started":1}
I20260809 07:53:36.774653 3820 tablet_service.cc:1467] Processing CreateTablet for tablet 302931c08cfc41318f677c70deddd5bd (DEFAULT_TABLE table=test-table [id=d950a5f01e9d4cb99e36a4ed4f67ea99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:36.775139 4025 tablet_service.cc:1467] Processing CreateTablet for tablet 302931c08cfc41318f677c70deddd5bd (DEFAULT_TABLE table=test-table [id=d950a5f01e9d4cb99e36a4ed4f67ea99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:36.775219 4224 tablet_service.cc:1467] Processing CreateTablet for tablet 302931c08cfc41318f677c70deddd5bd (DEFAULT_TABLE table=test-table [id=d950a5f01e9d4cb99e36a4ed4f67ea99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:36.776082 4224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302931c08cfc41318f677c70deddd5bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:36.776082 3820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302931c08cfc41318f677c70deddd5bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:36.776111 4025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302931c08cfc41318f677c70deddd5bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:36.781704 4367 tablet_bootstrap.cc:492] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc: Bootstrap starting.
I20260809 07:53:36.782778 4368 tablet_bootstrap.cc:492] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e: Bootstrap starting.
I20260809 07:53:36.783820 4371 tablet_bootstrap.cc:492] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34: Bootstrap starting.
I20260809 07:53:36.784219 4367 tablet_bootstrap.cc:654] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:36.784770 4368 tablet_bootstrap.cc:654] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:36.785071 4367 log.cc:824] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:36.785552 4368 log.cc:824] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:36.785668 4371 tablet_bootstrap.cc:654] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:36.786386 4371 log.cc:824] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:36.786980 4367 tablet_bootstrap.cc:492] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc: No bootstrap required, opened a new log
I20260809 07:53:36.787175 4367 ts_tablet_manager.cc:1397] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260809 07:53:36.788190 4368 tablet_bootstrap.cc:492] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e: No bootstrap required, opened a new log
I20260809 07:53:36.788364 4368 ts_tablet_manager.cc:1397] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 07:53:36.789361 4371 tablet_bootstrap.cc:492] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34: No bootstrap required, opened a new log
I20260809 07:53:36.789541 4371 ts_tablet_manager.cc:1397] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260809 07:53:36.791597 4367 raft_consensus.cc:348] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f6dfcaaa994aefa23a7429fd77397e" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40097 } } peers { permanent_uuid: "c180abd9f3bc4f37bd3287c64917b2cc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41715 } } peers { permanent_uuid: "8291fe35f1a14394a3d1101bac5a3c34" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 35181 } }
I20260809 07:53:36.791863 4367 raft_consensus.cc:374] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:36.791917 4367 raft_consensus.cc:729] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c180abd9f3bc4f37bd3287c64917b2cc, State: Initialized, Role: FOLLOWER
I20260809 07:53:36.792349 4367 consensus_queue.cc:260] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f6dfcaaa994aefa23a7429fd77397e" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40097 } } peers { permanent_uuid: "c180abd9f3bc4f37bd3287c64917b2cc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41715 } } peers { permanent_uuid: "8291fe35f1a14394a3d1101bac5a3c34" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 35181 } }
I20260809 07:53:36.792431 4368 raft_consensus.cc:348] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f6dfcaaa994aefa23a7429fd77397e" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40097 } } peers { permanent_uuid: "c180abd9f3bc4f37bd3287c64917b2cc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41715 } } peers { permanent_uuid: "8291fe35f1a14394a3d1101bac5a3c34" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 35181 } }
I20260809 07:53:36.792676 4368 raft_consensus.cc:374] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:36.792734 4368 raft_consensus.cc:729] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f6dfcaaa994aefa23a7429fd77397e, State: Initialized, Role: FOLLOWER
I20260809 07:53:36.792972 4371 raft_consensus.cc:348] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f6dfcaaa994aefa23a7429fd77397e" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40097 } } peers { permanent_uuid: "c180abd9f3bc4f37bd3287c64917b2cc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41715 } } peers { permanent_uuid: "8291fe35f1a14394a3d1101bac5a3c34" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 35181 } }
I20260809 07:53:36.793180 4371 raft_consensus.cc:374] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:36.793134 4368 consensus_queue.cc:260] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f6dfcaaa994aefa23a7429fd77397e" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40097 } } peers { permanent_uuid: "c180abd9f3bc4f37bd3287c64917b2cc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41715 } } peers { permanent_uuid: "8291fe35f1a14394a3d1101bac5a3c34" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 35181 } }
I20260809 07:53:36.793236 4371 raft_consensus.cc:729] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8291fe35f1a14394a3d1101bac5a3c34, State: Initialized, Role: FOLLOWER
I20260809 07:53:36.793341 4367 ts_tablet_manager.cc:1428] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260809 07:53:36.793596 4371 consensus_queue.cc:260] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f6dfcaaa994aefa23a7429fd77397e" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40097 } } peers { permanent_uuid: "c180abd9f3bc4f37bd3287c64917b2cc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41715 } } peers { permanent_uuid: "8291fe35f1a14394a3d1101bac5a3c34" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 35181 } }
I20260809 07:53:36.793787 4123 heartbeater.cc:503] Master 127.28.112.62:40329 was elected leader, sending a full tablet report...
I20260809 07:53:36.794056 4368 ts_tablet_manager.cc:1428] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 07:53:36.794157 3923 heartbeater.cc:503] Master 127.28.112.62:40329 was elected leader, sending a full tablet report...
I20260809 07:53:36.794526 4371 ts_tablet_manager.cc:1428] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 07:53:36.794843 4336 heartbeater.cc:503] Master 127.28.112.62:40329 was elected leader, sending a full tablet report...
W20260809 07:53:36.825373 4124 tablet.cc:2369] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 07:53:36.910912 3924 tablet.cc:2369] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 07:53:36.993580 4337 tablet.cc:2369] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:53:36.995998 4378 raft_consensus.cc:482] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:36.996210 4378 raft_consensus.cc:504] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f6dfcaaa994aefa23a7429fd77397e" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40097 } } peers { permanent_uuid: "c180abd9f3bc4f37bd3287c64917b2cc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41715 } } peers { permanent_uuid: "8291fe35f1a14394a3d1101bac5a3c34" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 35181 } }
I20260809 07:53:36.997355 4378 leader_election.cc:288] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c180abd9f3bc4f37bd3287c64917b2cc (127.28.112.2:41715), 8291fe35f1a14394a3d1101bac5a3c34 (127.28.112.3:35181)
I20260809 07:53:36.999536 4363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:36.997850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:51313 (local address 127.28.112.3:35181)
0809 07:53:36.997995 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:36.997999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:36.998175 (+ 176us) server_negotiation.cc:407] Connection header received
0809 07:53:36.998293 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:36.998297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:36.998337 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:36.998414 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:36.999156 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:36.999159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:36.999162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:53:36.999175 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:36.999199 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:53:36.999209 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:36.999284 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:36.999347 (+ 63us) server_negotiation.cc:299] Negotiation successful
0809 07:53:36.999388 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 07:53:36.999552 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:36.997668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:35181 (local address 127.28.112.1:51313)
0809 07:53:36.998067 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:36.998082 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:36.998195 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:36.998442 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:36.998444 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:36.998460 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:36.998970 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:53:36.998972 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:36.998984 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:53:36.999012 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:36.999270 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:36.999274 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:36.999276 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 07:53:36.999358 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 07:53:36.999396 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":95,"threads_started":1}
I20260809 07:53:36.999629 4365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:36.997774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:59549 (local address 127.28.112.2:41715)
0809 07:53:36.997911 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:36.997916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:36.998212 (+ 296us) server_negotiation.cc:407] Connection header received
0809 07:53:36.998397 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:36.998400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:36.998442 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:36.998521 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:36.999259 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:36.999263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:36.999266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:53:36.999279 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:36.999300 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:53:36.999311 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:36.999393 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:36.999480 (+ 87us) server_negotiation.cc:299] Negotiation successful
0809 07:53:36.999518 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 07:53:36.999779 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:36.997668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:41715 (local address 127.28.112.1:59549)
0809 07:53:36.998109 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:36.998120 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:36.998280 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:36.998572 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:36.998575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:36.998590 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:36.999119 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:53:36.999122 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:36.999131 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:53:36.999152 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:36.999397 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:36.999401 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:36.999404 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 07:53:36.999484 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 07:53:36.999517 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":30,"thread_start_us":62,"threads_started":1}
I20260809 07:53:37.000260 4260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302931c08cfc41318f677c70deddd5bd" candidate_uuid: "74f6dfcaaa994aefa23a7429fd77397e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8291fe35f1a14394a3d1101bac5a3c34" is_pre_election: true
I20260809 07:53:37.000425 4051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302931c08cfc41318f677c70deddd5bd" candidate_uuid: "74f6dfcaaa994aefa23a7429fd77397e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c180abd9f3bc4f37bd3287c64917b2cc" is_pre_election: true
I20260809 07:53:37.000595 4260 raft_consensus.cc:2454] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74f6dfcaaa994aefa23a7429fd77397e in term 0.
I20260809 07:53:37.000667 4051 raft_consensus.cc:2454] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74f6dfcaaa994aefa23a7429fd77397e in term 0.
I20260809 07:53:37.001066 3749 leader_election.cc:302] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74f6dfcaaa994aefa23a7429fd77397e, c180abd9f3bc4f37bd3287c64917b2cc; no voters:
I20260809 07:53:37.001328 4378 raft_consensus.cc:2781] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:53:37.001417 4378 raft_consensus.cc:482] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:53:37.001467 4378 raft_consensus.cc:3037] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:37.002547 4378 raft_consensus.cc:504] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f6dfcaaa994aefa23a7429fd77397e" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40097 } } peers { permanent_uuid: "c180abd9f3bc4f37bd3287c64917b2cc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41715 } } peers { permanent_uuid: "8291fe35f1a14394a3d1101bac5a3c34" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 35181 } }
I20260809 07:53:37.002864 4378 leader_election.cc:288] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e [CANDIDATE]: Term 1 election: Requested vote from peers c180abd9f3bc4f37bd3287c64917b2cc (127.28.112.2:41715), 8291fe35f1a14394a3d1101bac5a3c34 (127.28.112.3:35181)
I20260809 07:53:37.003224 4260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302931c08cfc41318f677c70deddd5bd" candidate_uuid: "74f6dfcaaa994aefa23a7429fd77397e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8291fe35f1a14394a3d1101bac5a3c34"
I20260809 07:53:37.003235 4051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302931c08cfc41318f677c70deddd5bd" candidate_uuid: "74f6dfcaaa994aefa23a7429fd77397e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c180abd9f3bc4f37bd3287c64917b2cc"
I20260809 07:53:37.003331 4260 raft_consensus.cc:3037] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:37.003373 4051 raft_consensus.cc:3037] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:37.004318 4260 raft_consensus.cc:2454] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74f6dfcaaa994aefa23a7429fd77397e in term 1.
I20260809 07:53:37.004432 4051 raft_consensus.cc:2454] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74f6dfcaaa994aefa23a7429fd77397e in term 1.
I20260809 07:53:37.004603 3747 leader_election.cc:302] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74f6dfcaaa994aefa23a7429fd77397e, 8291fe35f1a14394a3d1101bac5a3c34; no voters:
I20260809 07:53:37.004779 4378 raft_consensus.cc:2781] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:37.005002 4378 raft_consensus.cc:686] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e [term 1 LEADER]: Becoming Leader. State: Replica: 74f6dfcaaa994aefa23a7429fd77397e, State: Running, Role: LEADER
I20260809 07:53:37.005306 4378 consensus_queue.cc:237] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f6dfcaaa994aefa23a7429fd77397e" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40097 } } peers { permanent_uuid: "c180abd9f3bc4f37bd3287c64917b2cc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41715 } } peers { permanent_uuid: "8291fe35f1a14394a3d1101bac5a3c34" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 35181 } }
I20260809 07:53:37.008569 3617 catalog_manager.cc:5495] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e reported cstate change: term changed from 0 to 1, leader changed from <none> to 74f6dfcaaa994aefa23a7429fd77397e (127.28.112.1). New cstate: current_term: 1 leader_uuid: "74f6dfcaaa994aefa23a7429fd77397e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f6dfcaaa994aefa23a7429fd77397e" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c180abd9f3bc4f37bd3287c64917b2cc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8291fe35f1a14394a3d1101bac5a3c34" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 35181 } health_report { overall_health: UNKNOWN } } }
I20260809 07:53:37.061589 4361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:37.060285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44128 (local address 127.28.112.1:40097)
0809 07:53:37.060401 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:37.060404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:37.060424 (+ 20us) server_negotiation.cc:407] Connection header received
0809 07:53:37.060517 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:37.060520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:37.060580 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:37.060651 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:37.061263 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:37.061266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:37.061268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:53:37.061280 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:37.061309 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:53:37.061318 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:37.061390 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:37.061450 (+ 60us) server_negotiation.cc:299] Negotiation successful
0809 07:53:37.061485 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 07:53:37.067351 4051 raft_consensus.cc:1261] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc [term 1 FOLLOWER]: Refusing update from remote peer 74f6dfcaaa994aefa23a7429fd77397e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:37.067361 4260 raft_consensus.cc:1261] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34 [term 1 FOLLOWER]: Refusing update from remote peer 74f6dfcaaa994aefa23a7429fd77397e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:37.068043 4384 consensus_queue.cc:1035] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8291fe35f1a14394a3d1101bac5a3c34" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 35181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:37.068217 4378 consensus_queue.cc:1035] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c180abd9f3bc4f37bd3287c64917b2cc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:37.078953 4392 mvcc.cc:205] Tried to move back new op lower bound from 7316529221900967936 to 7316529221656145920. Current Snapshot: MvccSnapshot[applied={T|T < 7316529221900967936}]
I20260809 07:53:37.084131 3618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:37.084275 3618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:37.086521 3618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e66a91921ab744db81708c6a2991b1a9: Sending DeleteTablet for 3 replicas of tablet 302931c08cfc41318f677c70deddd5bd
I20260809 07:53:37.087234 4025 tablet_service.cc:1514] Processing DeleteTablet for tablet 302931c08cfc41318f677c70deddd5bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:37 UTC) from {username='slave'} at 127.0.0.1:34506
I20260809 07:53:37.087232 3820 tablet_service.cc:1514] Processing DeleteTablet for tablet 302931c08cfc41318f677c70deddd5bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:37 UTC) from {username='slave'} at 127.0.0.1:44116
I20260809 07:53:37.087697 4407 tablet_replica.cc:330] T 302931c08cfc41318f677c70deddd5bd P 74f6dfcaaa994aefa23a7429fd77397e: stopping tablet replica
I20260809 07:53:37.087725 4224 tablet_service.cc:1514] Processing DeleteTablet for tablet 302931c08cfc41318f677c70deddd5bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:37 UTC) from {username='slave'} at 127.0.0.1:58942
I20260809 07:53:37.087966 4405 tablet_replica.cc:330] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc: stopping tablet replica
I20260809 07:53:37.088160 4408 tablet_replica.cc:330] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34: stopping tablet replica
I20260809 07:53:37.088264 4405 raft_consensus.cc:2229] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:37.088415 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 3722
I20260809 07:53:37.088471 4408 raft_consensus.cc:2229] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:37.088598 4405 raft_consensus.cc:2258] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:37.088723 4408 raft_consensus.cc:2258] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:37.093255 4408 ts_tablet_manager.cc:1905] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:37.093255 4405 ts_tablet_manager.cc:1905] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:37.095944 4405 ts_tablet_manager.cc:1918] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:53:37.095944 4408 ts_tablet_manager.cc:1918] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:53:37.096036 4405 log.cc:1196] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/302931c08cfc41318f677c70deddd5bd
I20260809 07:53:37.096036 4408 log.cc:1196] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/302931c08cfc41318f677c70deddd5bd
I20260809 07:53:37.096350 4408 ts_tablet_manager.cc:1939] T 302931c08cfc41318f677c70deddd5bd P 8291fe35f1a14394a3d1101bac5a3c34: Deleting consensus metadata
I20260809 07:53:37.096354 4405 ts_tablet_manager.cc:1939] T 302931c08cfc41318f677c70deddd5bd P c180abd9f3bc4f37bd3287c64917b2cc: Deleting consensus metadata
I20260809 07:53:37.097573 3602 catalog_manager.cc:4841] TS c180abd9f3bc4f37bd3287c64917b2cc (127.28.112.2:41715): tablet 302931c08cfc41318f677c70deddd5bd (table test-table [id=d950a5f01e9d4cb99e36a4ed4f67ea99]) successfully deleted
I20260809 07:53:37.097573 3600 catalog_manager.cc:4841] TS 8291fe35f1a14394a3d1101bac5a3c34 (127.28.112.3:35181): tablet 302931c08cfc41318f677c70deddd5bd (table test-table [id=d950a5f01e9d4cb99e36a4ed4f67ea99]) successfully deleted
W20260809 07:53:37.107333 3599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.112.1:40097 (error 108)
I20260809 07:53:37.107553 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 3926
I20260809 07:53:37.108793 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:37.108222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:40097 (local address 127.0.0.1:44138)
0809 07:53:37.108632 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:37.108706 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:40097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311}
W20260809 07:53:37.110386 3599 catalog_manager.cc:4584] TS 74f6dfcaaa994aefa23a7429fd77397e (127.28.112.1:40097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 302931c08cfc41318f677c70deddd5bd: Network error: Client connection negotiation failed: client connection to 127.28.112.1:40097: connect: Connection refused (error 111)
I20260809 07:53:37.115808 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 4126
I20260809 07:53:37.125311 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 3577
2026-08-09T07:53:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1163 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260809 07:53:37.151610 29120 mini_oidc.cc:152] Starting JWKS server
I20260809 07:53:37.153170 29120 webserver.cc:460] Webserver started at https://127.0.0.1:42895/ using document root <none> and password file <none>
I20260809 07:53:37.153317 29120 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 07:53:37.153548 29120 webserver.cc:460] Webserver started at http://127.0.0.1:43947/ using document root <none> and password file <none>
2026-08-09T07:53:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:53:37Z Disabled control of system clock
I20260809 07:53:37.162405 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:41093
--webserver_interface=127.28.112.62
--webserver_port=0
--builtin_ntp_servers=127.28.112.20:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:41093
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42895/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0/testchainca.cert with env {}
W20260809 07:53:37.270018 4417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:37.270258 4417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:37.270313 4417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:37.274184 4417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:37.274232 4417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:37.274252 4417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:37.274274 4417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:37.274302 4417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 07:53:37.274325 4417 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 07:53:37.274353 4417 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42895/jwks.json
I20260809 07:53:37.279471 4417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:41093
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:41093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42895/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:37.280381 4417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:37.281296 4417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:37.289073 4426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:37.289080 4422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:37.289258 4417 server_base.cc:1034] running on GCE node
W20260809 07:53:37.289080 4423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:37.289901 4417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:37.290948 4417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:37.292102 4417 hybrid_clock.cc:648] HybridClock initialized: now 1786262017292088 us; error 27 us; skew 500 ppm
I20260809 07:53:37.293824 4417 webserver.cc:460] Webserver started at http://127.28.112.62:44459/ using document root <none> and password file <none>
I20260809 07:53:37.294418 4417 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:37.294486 4417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:37.294668 4417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:37.296584 4417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "12f5a07a43ea4e3a9c4d3da96dbc8f45"
format_stamp: "Formatted at 2026-08-09 07:53:37 on dist-test-slave-vpvm"
server_key: "2c955ece9c989e7c14d063c6d376c9ad"
server_key_iv: "1fe1d54071e1bd4d3ea35e258cbb6225"
server_key_version: "encryptionkey@0"
I20260809 07:53:37.297106 4417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "12f5a07a43ea4e3a9c4d3da96dbc8f45"
format_stamp: "Formatted at 2026-08-09 07:53:37 on dist-test-slave-vpvm"
server_key: "2c955ece9c989e7c14d063c6d376c9ad"
server_key_iv: "1fe1d54071e1bd4d3ea35e258cbb6225"
server_key_version: "encryptionkey@0"
I20260809 07:53:37.300581 4417 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260809 07:53:37.302933 4436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:37.304149 4417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:37.304263 4417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "12f5a07a43ea4e3a9c4d3da96dbc8f45"
format_stamp: "Formatted at 2026-08-09 07:53:37 on dist-test-slave-vpvm"
server_key: "2c955ece9c989e7c14d063c6d376c9ad"
server_key_iv: "1fe1d54071e1bd4d3ea35e258cbb6225"
server_key_version: "encryptionkey@0"
I20260809 07:53:37.304355 4417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:37.319227 4417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:37.319860 4417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:37.320031 4417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:37.326409 4417 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:41093
I20260809 07:53:37.326437 4518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:41093 every 8 connection(s)
I20260809 07:53:37.327312 4417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:53:37.330411 4521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:37.335821 4521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45: Bootstrap starting.
I20260809 07:53:37.337213 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 4417
I20260809 07:53:37.337333 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:53:37.337632 29120 external_mini_cluster.cc:1428] Setting key 06bf74e4b6b2b4563efa49ecf95ce387
I20260809 07:53:37.338639 4521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:37.339463 4521 log.cc:824] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:37.341399 4521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45: No bootstrap required, opened a new log
I20260809 07:53:37.344871 4521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f5a07a43ea4e3a9c4d3da96dbc8f45" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 41093 } }
I20260809 07:53:37.345084 4521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:37.345118 4521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f5a07a43ea4e3a9c4d3da96dbc8f45, State: Initialized, Role: FOLLOWER
I20260809 07:53:37.345608 4521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f5a07a43ea4e3a9c4d3da96dbc8f45" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 41093 } }
I20260809 07:53:37.345782 4526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:37.345907 4526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f5a07a43ea4e3a9c4d3da96dbc8f45" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 41093 } }
I20260809 07:53:37.345755 4521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:53:37.346271 4521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:53:37.346251 4526 leader_election.cc:302] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12f5a07a43ea4e3a9c4d3da96dbc8f45; no voters:
I20260809 07:53:37.346371 4521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:37.346629 4526 leader_election.cc:288] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260809 07:53:37.346654 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:37.338725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60610 (local address 127.28.112.62:41093)
0809 07:53:37.339374 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:37.339388 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:37.339422 (+ 34us) server_negotiation.cc:407] Connection header received
0809 07:53:37.340160 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:37.340185 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:37.340542 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:37.340853 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:37.341276 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:37.342292 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:37.343176 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:37.343664 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:37.343813 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:37.344643 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:37.344649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:37.344662 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:53:37.345042 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:37.345113 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:53:37.345144 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:37.345271 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:37.345407 (+ 136us) server_negotiation.cc:299] Negotiation successful
0809 07:53:37.345701 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":132,"threads_started":1}
I20260809 07:53:37.347483 4521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f5a07a43ea4e3a9c4d3da96dbc8f45" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 41093 } }
I20260809 07:53:37.347620 4521 leader_election.cc:302] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12f5a07a43ea4e3a9c4d3da96dbc8f45; no voters:
I20260809 07:53:37.347659 4528 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260809 07:53:37.347707 4521 leader_election.cc:288] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:53:37.347981 4526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:37.348239 4526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [term 1 LEADER]: Becoming Leader. State: Replica: 12f5a07a43ea4e3a9c4d3da96dbc8f45, State: Running, Role: LEADER
I20260809 07:53:37.348570 4526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f5a07a43ea4e3a9c4d3da96dbc8f45" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 41093 } }
I20260809 07:53:37.348994 4521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:37.349910 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:53:37.349857 4528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12f5a07a43ea4e3a9c4d3da96dbc8f45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f5a07a43ea4e3a9c4d3da96dbc8f45" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 41093 } } }
I20260809 07:53:37.350001 4528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:37.350376 4540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:53:37.351562 4530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12f5a07a43ea4e3a9c4d3da96dbc8f45. Latest consensus state: current_term: 1 leader_uuid: "12f5a07a43ea4e3a9c4d3da96dbc8f45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f5a07a43ea4e3a9c4d3da96dbc8f45" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 41093 } } }
I20260809 07:53:37.351650 4530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:37.353134 4540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:37.358752 4540 catalog_manager.cc:1347] Generated new cluster ID: 909117162830452a84300c701aa50143
I20260809 07:53:37.358800 4540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:37.373893 4540 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:53:37.374722 4540 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:37.380394 4540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12f5a07a43ea4e3a9c4d3da96dbc8f45: Generated new TSK 0
I20260809 07:53:37.381088 4540 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 07:53:37.414985 4560 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260809 07:53:37.415112 4560 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 07:53:37.415740 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:37.398190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60614 (local address 127.28.112.62:41093)
0809 07:53:37.398319 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:37.398325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:37.398516 (+ 191us) server_negotiation.cc:407] Connection header received
0809 07:53:37.398700 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:37.398704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:37.398764 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:37.398881 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0809 07:53:37.399136 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:37.399958 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:37.400838 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:37.401199 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:37.401314 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:37.401656 (+ 342us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 07:53:37.415324 (+ 13668us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 07:53:37.415591 (+ 267us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60614: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260809 07:53:37.415822 4524 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60614: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260809 07:53:37.416874 4559 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:37.398067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:41093 (local address 127.0.0.1:60614)
0809 07:53:37.398438 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:37.398451 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:37.398559 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:37.398919 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:37.398923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:37.398932 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0809 07:53:37.399035 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:37.399040 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:37.400090 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:37.400093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:37.400728 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:37.400736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:37.401331 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:37.401334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:37.401444 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:37.401489 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 07:53:37.415516 (+ 14027us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 07:53:37.415693 (+ 177us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.112.62:41093: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":124,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.112.62:41093: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260809 07:53:37.417765 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 4417
2026-08-09T07:53:37Z chronyd exiting
I20260809 07:53:37.458815 29120 test_util.cc:175] -----------------------------------------------
I20260809 07:53:37.458900 29120 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786261991120089-29120-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (308 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260809 07:53:37.460033 29120 mini_oidc.cc:152] Starting JWKS server
I20260809 07:53:37.460827 29120 webserver.cc:460] Webserver started at https://127.0.0.1:39471/ using document root <none> and password file <none>
I20260809 07:53:37.460932 29120 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 07:53:37.461189 29120 webserver.cc:460] Webserver started at http://127.0.0.1:45539/ using document root <none> and password file <none>
2026-08-09T07:53:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:53:37Z Disabled control of system clock
I20260809 07:53:37.469771 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:44361
--webserver_interface=127.28.112.62
--webserver_port=0
--builtin_ntp_servers=127.28.112.20:36993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:44361
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39471/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786261991120089-29120-0/testchainca.cert with env {}
W20260809 07:53:37.575820 4569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:37.576076 4569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:37.576117 4569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:37.579998 4569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:37.580045 4569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:37.580060 4569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:37.580088 4569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:37.580116 4569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 07:53:37.580143 4569 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 07:53:37.580169 4569 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39471/jwks.json
I20260809 07:53:37.585259 4569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:44361
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:44361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39471/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786261991120089-29120-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:37.586236 4569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:37.587163 4569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:37.595733 4576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:37.595726 4578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:37.595813 4575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:37.595937 4569 server_base.cc:1034] running on GCE node
I20260809 07:53:37.596547 4569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:37.597522 4569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:37.598672 4569 hybrid_clock.cc:648] HybridClock initialized: now 1786262017598651 us; error 39 us; skew 500 ppm
I20260809 07:53:37.600414 4569 webserver.cc:460] Webserver started at http://127.28.112.62:36457/ using document root <none> and password file <none>
I20260809 07:53:37.600943 4569 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:37.601009 4569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:37.601199 4569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:37.603179 4569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "36291122ca594e9ab72c69af71932689"
format_stamp: "Formatted at 2026-08-09 07:53:37 on dist-test-slave-vpvm"
server_key: "d41678bd8ec3cf14d62c21ac8344a165"
server_key_iv: "fbea352b97e3f01641395bdcff5d1712"
server_key_version: "encryptionkey@0"
I20260809 07:53:37.603704 4569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "36291122ca594e9ab72c69af71932689"
format_stamp: "Formatted at 2026-08-09 07:53:37 on dist-test-slave-vpvm"
server_key: "d41678bd8ec3cf14d62c21ac8344a165"
server_key_iv: "fbea352b97e3f01641395bdcff5d1712"
server_key_version: "encryptionkey@0"
I20260809 07:53:37.607122 4569 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:53:37.609340 4590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:37.610546 4569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:37.610647 4569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "36291122ca594e9ab72c69af71932689"
format_stamp: "Formatted at 2026-08-09 07:53:37 on dist-test-slave-vpvm"
server_key: "d41678bd8ec3cf14d62c21ac8344a165"
server_key_iv: "fbea352b97e3f01641395bdcff5d1712"
server_key_version: "encryptionkey@0"
I20260809 07:53:37.610728 4569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:37.625191 4569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:37.625757 4569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:37.625895 4569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:37.632386 4569 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:44361
I20260809 07:53:37.632405 4664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:44361 every 8 connection(s)
I20260809 07:53:37.633279 4569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:53:37.634359 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 4569
I20260809 07:53:37.634452 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:53:37.634716 29120 external_mini_cluster.cc:1428] Setting key fe3c5297a4e9e53efc060b86a96e8b4f
I20260809 07:53:37.636790 4665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:37.642055 4665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689: Bootstrap starting.
I20260809 07:53:37.643330 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:37.636008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49558 (local address 127.28.112.62:44361)
0809 07:53:37.636624 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:37.636650 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:37.636685 (+ 35us) server_negotiation.cc:407] Connection header received
0809 07:53:37.637334 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:37.637352 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:37.637736 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:37.638062 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:37.638474 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:37.639416 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:37.640293 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:37.640751 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:37.640895 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:37.641694 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:37.641699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:37.641711 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:53:37.642109 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:37.642167 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:53:37.642195 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:37.642283 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:37.642411 (+ 128us) server_negotiation.cc:299] Negotiation successful
0809 07:53:37.642662 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":145,"threads_started":1}
I20260809 07:53:37.644212 4665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:37.644914 4665 log.cc:824] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:37.645993 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:53:37.646898 4665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689: No bootstrap required, opened a new log
I20260809 07:53:37.650256 4665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36291122ca594e9ab72c69af71932689" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 44361 } }
I20260809 07:53:37.650466 4665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:37.650525 4665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36291122ca594e9ab72c69af71932689, State: Initialized, Role: FOLLOWER
I20260809 07:53:37.650955 4665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36291122ca594e9ab72c69af71932689" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 44361 } }
I20260809 07:53:37.651119 4665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:53:37.651188 4665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:53:37.651285 4665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:37.652101 4665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36291122ca594e9ab72c69af71932689" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 44361 } }
I20260809 07:53:37.652420 4665 leader_election.cc:302] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36291122ca594e9ab72c69af71932689; no voters:
I20260809 07:53:37.652686 4665 leader_election.cc:288] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:53:37.652766 4671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:37.652985 4671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689 [term 1 LEADER]: Becoming Leader. State: Replica: 36291122ca594e9ab72c69af71932689, State: Running, Role: LEADER
I20260809 07:53:37.653292 4671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36291122ca594e9ab72c69af71932689" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 44361 } }
I20260809 07:53:37.653851 4665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:37.654834 4675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36291122ca594e9ab72c69af71932689. Latest consensus state: current_term: 1 leader_uuid: "36291122ca594e9ab72c69af71932689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36291122ca594e9ab72c69af71932689" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 44361 } } }
I20260809 07:53:37.654964 4675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:37.655246 4674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36291122ca594e9ab72c69af71932689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36291122ca594e9ab72c69af71932689" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 44361 } } }
I20260809 07:53:37.655333 4686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:53:37.655337 4674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:37.657931 4686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:37.663575 4686 catalog_manager.cc:1347] Generated new cluster ID: 021895a2188442ab921a7aef21d84875
I20260809 07:53:37.663627 4686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:37.675437 4686 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:53:37.676222 4686 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:37.680630 4686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36291122ca594e9ab72c69af71932689: Generated new TSK 0
I20260809 07:53:37.681186 4686 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 07:53:37.707005 4707 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260809 07:53:37.707073 4707 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 07:53:37.707749 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:37.692370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.28.112.62:44361)
0809 07:53:37.692492 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:37.692497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:37.692648 (+ 151us) server_negotiation.cc:407] Connection header received
0809 07:53:37.692791 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:37.692795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:37.692847 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:37.692950 (+ 103us) server_negotiation.cc:226] Negotiated authn=JWT
0809 07:53:37.693139 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:37.693991 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:37.694834 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:37.695162 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:37.695258 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:37.695529 (+ 271us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 07:53:37.707370 (+ 11841us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 07:53:37.707618 (+ 248us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49570: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":49}
W20260809 07:53:37.707789 4706 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:37.692274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:44361 (local address 127.0.0.1:49570)
0809 07:53:37.692571 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:37.692584 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:37.692681 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:37.692952 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:37.692954 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:37.692960 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0809 07:53:37.693057 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:37.693062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:37.694084 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:37.694086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:37.694718 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:37.694727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:37.695262 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:37.695264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:37.695324 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:37.695347 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 07:53:37.707535 (+ 12188us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 07:53:37.707658 (+ 123us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.112.62:44361: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":123,"threads_started":1}
W20260809 07:53:37.707813 4669 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49570: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260809 07:53:37.708478 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 4569
2026-08-09T07:53:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (295 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260809 07:53:37.755043 29120 mini_oidc.cc:152] Starting JWKS server
I20260809 07:53:37.755854 29120 webserver.cc:460] Webserver started at https://127.0.0.1:41773/ using document root <none> and password file <none>
I20260809 07:53:37.755944 29120 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 07:53:37.756186 29120 webserver.cc:460] Webserver started at http://127.0.0.1:35171/ using document root <none> and password file <none>
2026-08-09T07:53:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:53:37Z Disabled control of system clock
I20260809 07:53:37.765410 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:34393
--webserver_interface=127.28.112.62
--webserver_port=0
--builtin_ntp_servers=127.28.112.20:39083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:34393
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41773/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 07:53:37.871953 4715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:37.872223 4715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:37.872265 4715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:37.876207 4715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:37.876268 4715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:37.876283 4715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:37.876313 4715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:37.876339 4715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 07:53:37.876366 4715 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 07:53:37.876397 4715 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41773/jwks.json
I20260809 07:53:37.881084 4715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:39083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:34393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:34393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41773/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:37.882154 4715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:37.883520 4715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:37.891608 4727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:37.891629 4729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:37.891633 4726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:37.891846 4715 server_base.cc:1034] running on GCE node
I20260809 07:53:37.892421 4715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:37.893486 4715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:37.894654 4715 hybrid_clock.cc:648] HybridClock initialized: now 1786262017894640 us; error 31 us; skew 500 ppm
I20260809 07:53:37.896301 4715 webserver.cc:460] Webserver started at http://127.28.112.62:41685/ using document root <none> and password file <none>
I20260809 07:53:37.896852 4715 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:37.896914 4715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:37.897109 4715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:37.899080 4715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "4d3c610f90e4447f8d6058131a871a34"
format_stamp: "Formatted at 2026-08-09 07:53:37 on dist-test-slave-vpvm"
server_key: "29d73bfea01754a8d9b8e345b47e4910"
server_key_iv: "4ed2f4627256525a45e18a375f9a0786"
server_key_version: "encryptionkey@0"
I20260809 07:53:37.899596 4715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "4d3c610f90e4447f8d6058131a871a34"
format_stamp: "Formatted at 2026-08-09 07:53:37 on dist-test-slave-vpvm"
server_key: "29d73bfea01754a8d9b8e345b47e4910"
server_key_iv: "4ed2f4627256525a45e18a375f9a0786"
server_key_version: "encryptionkey@0"
I20260809 07:53:37.903047 4715 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260809 07:53:37.905412 4740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:37.906682 4715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:37.906788 4715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "4d3c610f90e4447f8d6058131a871a34"
format_stamp: "Formatted at 2026-08-09 07:53:37 on dist-test-slave-vpvm"
server_key: "29d73bfea01754a8d9b8e345b47e4910"
server_key_iv: "4ed2f4627256525a45e18a375f9a0786"
server_key_version: "encryptionkey@0"
I20260809 07:53:37.906901 4715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:37.948864 4715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:37.949574 4715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:37.949745 4715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:37.956429 4715 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:34393
I20260809 07:53:37.956447 4809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:34393 every 8 connection(s)
I20260809 07:53:37.957373 4715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:53:37.960213 4811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:37.960301 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 4715
I20260809 07:53:37.960408 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:53:37.960724 29120 external_mini_cluster.cc:1428] Setting key 03fd11d48a3d7e82f392c96f9e54633a
I20260809 07:53:37.966496 4811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34: Bootstrap starting.
I20260809 07:53:37.968612 4811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:37.969310 4816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:37.962075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.28.112.62:34393)
0809 07:53:37.962692 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:37.962706 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:37.962741 (+ 35us) server_negotiation.cc:407] Connection header received
0809 07:53:37.963444 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:37.963461 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:37.963828 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:37.964135 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:37.964536 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:37.965460 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:37.966360 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:37.966843 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:37.966997 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:37.967745 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:37.967750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:37.967762 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:53:37.968120 (+ 358us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:37.968173 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:53:37.968192 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:37.968276 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:37.968412 (+ 136us) server_negotiation.cc:299] Negotiation successful
0809 07:53:37.968642 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":121,"threads_started":1}
I20260809 07:53:37.969307 4811 log.cc:824] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:37.971220 4811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34: No bootstrap required, opened a new log
I20260809 07:53:37.971952 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:53:37.975021 4811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c610f90e4447f8d6058131a871a34" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 34393 } }
I20260809 07:53:37.975225 4811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:37.975267 4811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3c610f90e4447f8d6058131a871a34, State: Initialized, Role: FOLLOWER
I20260809 07:53:37.975672 4811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c610f90e4447f8d6058131a871a34" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 34393 } }
I20260809 07:53:37.975795 4811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:53:37.975836 4811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:53:37.975929 4811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:37.976894 4811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c610f90e4447f8d6058131a871a34" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 34393 } }
I20260809 07:53:37.977244 4811 leader_election.cc:302] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d3c610f90e4447f8d6058131a871a34; no voters:
I20260809 07:53:37.977571 4811 leader_election.cc:288] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:53:37.977661 4818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:37.977859 4818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3c610f90e4447f8d6058131a871a34, State: Running, Role: LEADER
I20260809 07:53:37.978179 4818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c610f90e4447f8d6058131a871a34" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 34393 } }
I20260809 07:53:37.978777 4811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:37.979808 4824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d3c610f90e4447f8d6058131a871a34. Latest consensus state: current_term: 1 leader_uuid: "4d3c610f90e4447f8d6058131a871a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c610f90e4447f8d6058131a871a34" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 34393 } } }
I20260809 07:53:37.979831 4820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d3c610f90e4447f8d6058131a871a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c610f90e4447f8d6058131a871a34" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 34393 } } }
I20260809 07:53:37.980017 4820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:37.980051 4824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:37.980748 4836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:53:37.983346 4836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:37.988497 4836 catalog_manager.cc:1347] Generated new cluster ID: 9b4216294fa641f89d77f55a37a193dc
I20260809 07:53:37.988549 4836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:37.998678 4836 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:53:37.999516 4836 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:38.010033 4836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d3c610f90e4447f8d6058131a871a34: Generated new TSK 0
I20260809 07:53:38.010720 4836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:53:38.025748 4816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:38.020953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.28.112.62:34393)
0809 07:53:38.021070 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:38.021075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:38.021196 (+ 121us) server_negotiation.cc:407] Connection header received
0809 07:53:38.021346 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:38.021348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:38.021391 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:38.021475 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0809 07:53:38.021729 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:38.022310 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:38.023553 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:38.023776 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:38.023856 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:38.024148 (+ 292us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 07:53:38.025400 (+ 1252us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41773/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 07:53:38.025617 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46960: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41773/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":48}
W20260809 07:53:38.025806 4816 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46960: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41773/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260809 07:53:38.027644 4857 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:38.020816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:34393 (local address 127.0.0.1:46960)
0809 07:53:38.021112 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:38.021126 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:38.021242 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:38.021494 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:38.021497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:38.021506 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0809 07:53:38.021630 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:38.021648 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:38.022411 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:38.022415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:38.023422 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:38.023433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:38.023891 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:38.023895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:38.023983 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:38.024011 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 07:53:38.025559 (+ 1548us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41773/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 07:53:38.027464 (+ 1905us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.112.62:34393: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41773/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":1708,"thread_start_us":109,"threads_started":1}
I20260809 07:53:38.028550 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 4715
E20260809 07:53:38.038069 4858 webserver.cc:533] Webserver: sslize failed:
E20260809 07:53:38.038138 4858 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-09T07:53:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (318 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-09T07:53:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:53:38Z Disabled control of system clock
I20260809 07:53:38.082373 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:35941
--webserver_interface=127.28.112.62
--webserver_port=0
--builtin_ntp_servers=127.28.112.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:35941
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786261991120089-29120-0/testchainca.cert with env {}
W20260809 07:53:38.189710 4863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:38.190021 4863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:38.190074 4863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:38.194079 4863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:38.194135 4863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:38.194149 4863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:38.194164 4863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:38.194178 4863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 07:53:38.194197 4863 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 07:53:38.194224 4863 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260809 07:53:38.199438 4863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:35941
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:35941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786261991120089-29120-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:38.200388 4863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:38.201339 4863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:38.209259 4868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:38.209352 4870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:38.209381 4872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:38.209424 4863 server_base.cc:1034] running on GCE node
I20260809 07:53:38.210147 4863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:38.211143 4863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:38.212294 4863 hybrid_clock.cc:648] HybridClock initialized: now 1786262018212278 us; error 28 us; skew 500 ppm
I20260809 07:53:38.214046 4863 webserver.cc:460] Webserver started at http://127.28.112.62:37855/ using document root <none> and password file <none>
I20260809 07:53:38.214579 4863 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:38.214638 4863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:38.214823 4863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:38.216732 4863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "be18b16fa75146e78c2912f74e218ce6"
format_stamp: "Formatted at 2026-08-09 07:53:38 on dist-test-slave-vpvm"
server_key: "c40e2ba30968c3faed78f6897180704a"
server_key_iv: "9dc8dabf761088884110a43ca52d8b81"
server_key_version: "encryptionkey@0"
I20260809 07:53:38.217239 4863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "be18b16fa75146e78c2912f74e218ce6"
format_stamp: "Formatted at 2026-08-09 07:53:38 on dist-test-slave-vpvm"
server_key: "c40e2ba30968c3faed78f6897180704a"
server_key_iv: "9dc8dabf761088884110a43ca52d8b81"
server_key_version: "encryptionkey@0"
I20260809 07:53:38.220609 4863 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:53:38.222877 4884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:38.223996 4863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:38.224107 4863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "be18b16fa75146e78c2912f74e218ce6"
format_stamp: "Formatted at 2026-08-09 07:53:38 on dist-test-slave-vpvm"
server_key: "c40e2ba30968c3faed78f6897180704a"
server_key_iv: "9dc8dabf761088884110a43ca52d8b81"
server_key_version: "encryptionkey@0"
I20260809 07:53:38.224193 4863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:38.243383 4863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:38.244020 4863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:38.244190 4863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:38.251412 4863 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:35941
I20260809 07:53:38.251425 4967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:35941 every 8 connection(s)
I20260809 07:53:38.252630 4863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:53:38.256047 4968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:38.257452 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 4863
I20260809 07:53:38.257576 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:53:38.257890 29120 external_mini_cluster.cc:1428] Setting key ee2401892342e9d0c752dca35baa5a60
I20260809 07:53:38.262368 4968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6: Bootstrap starting.
I20260809 07:53:38.264691 4968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:38.265515 4968 log.cc:824] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:38.266847 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:38.259348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39718 (local address 127.28.112.62:35941)
0809 07:53:38.260003 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:38.260017 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:38.260056 (+ 39us) server_negotiation.cc:407] Connection header received
0809 07:53:38.260662 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:38.260678 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:38.261056 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:38.261358 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:38.261798 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:38.262821 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:38.263699 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:38.264164 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:38.264318 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:38.265120 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:38.265125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:38.265140 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:53:38.265519 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:38.265601 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:53:38.265631 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:38.265727 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:38.265863 (+ 136us) server_negotiation.cc:299] Negotiation successful
0809 07:53:38.266139 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":144,"threads_started":1}
I20260809 07:53:38.267565 4968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6: No bootstrap required, opened a new log
I20260809 07:53:38.269495 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:53:38.271414 4968 raft_consensus.cc:348] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be18b16fa75146e78c2912f74e218ce6" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 35941 } }
I20260809 07:53:38.271633 4968 raft_consensus.cc:374] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:38.271675 4968 raft_consensus.cc:729] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be18b16fa75146e78c2912f74e218ce6, State: Initialized, Role: FOLLOWER
I20260809 07:53:38.272035 4968 consensus_queue.cc:260] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be18b16fa75146e78c2912f74e218ce6" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 35941 } }
I20260809 07:53:38.272158 4968 raft_consensus.cc:388] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:53:38.272202 4968 raft_consensus.cc:482] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:53:38.272271 4968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:38.273089 4968 raft_consensus.cc:504] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be18b16fa75146e78c2912f74e218ce6" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 35941 } }
I20260809 07:53:38.273518 4968 leader_election.cc:302] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be18b16fa75146e78c2912f74e218ce6; no voters:
I20260809 07:53:38.273816 4968 leader_election.cc:288] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:53:38.273917 4977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:38.274150 4977 raft_consensus.cc:686] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6 [term 1 LEADER]: Becoming Leader. State: Replica: be18b16fa75146e78c2912f74e218ce6, State: Running, Role: LEADER
I20260809 07:53:38.274446 4977 consensus_queue.cc:237] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be18b16fa75146e78c2912f74e218ce6" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 35941 } }
I20260809 07:53:38.275051 4968 sys_catalog.cc:564] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:38.275909 4978 sys_catalog.cc:455] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be18b16fa75146e78c2912f74e218ce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be18b16fa75146e78c2912f74e218ce6" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 35941 } } }
I20260809 07:53:38.276036 4978 sys_catalog.cc:458] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:38.276525 4980 sys_catalog.cc:455] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be18b16fa75146e78c2912f74e218ce6. Latest consensus state: current_term: 1 leader_uuid: "be18b16fa75146e78c2912f74e218ce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be18b16fa75146e78c2912f74e218ce6" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 35941 } } }
I20260809 07:53:38.276641 4980 sys_catalog.cc:458] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:38.276675 4991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:53:38.279337 4991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:38.284471 4991 catalog_manager.cc:1347] Generated new cluster ID: 677b3cc7efdb451ebb68035c6997ef7b
I20260809 07:53:38.284524 4991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:38.301705 4991 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:53:38.302636 4991 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:38.311811 4991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be18b16fa75146e78c2912f74e218ce6: Generated new TSK 0
I20260809 07:53:38.312456 4991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:53:38.808841 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:38.321608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39724 (local address 127.28.112.62:35941)
0809 07:53:38.321735 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:38.321740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:38.321897 (+ 157us) server_negotiation.cc:407] Connection header received
0809 07:53:38.322132 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:38.322135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:38.322184 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:38.322276 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0809 07:53:38.322573 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:38.323131 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:38.324381 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:38.324617 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:38.324700 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:38.325042 (+ 342us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 07:53:38.808452 (+483410us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 07:53:38.808721 (+ 269us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39724: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":55}
W20260809 07:53:38.808919 4975 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39724: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260809 07:53:38.809027 5016 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:38.321460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:35941 (local address 127.0.0.1:39724)
0809 07:53:38.321801 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:38.321818 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:38.321989 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:38.322290 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:38.322293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:38.322303 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0809 07:53:38.322436 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:38.322444 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:38.323251 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:38.323255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:38.324245 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:38.324258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:38.324737 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:38.324741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:38.324841 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:38.324873 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 07:53:38.808691 (+483818us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 07:53:38.808866 (+ 175us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.112.62:35941: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":135,"threads_started":1}
I20260809 07:53:38.809738 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 4863
2026-08-09T07:53:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (757 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (120 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:53:39 dist-test-slave-vpvm krb5kdc[5033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:53:39 dist-test-slave-vpvm krb5kdc[5033](info): set up 2 sockets
Aug 09 07:53:39 dist-test-slave-vpvm krb5kdc[5033](info): commencing operation
krb5kdc: starting...
W20260809 07:53:40.176477 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:53:40 dist-test-slave-vpvm krb5kdc[5033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:53:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:53:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:53:40.294126 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:42185
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:42185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:40.404994 5057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:40.405241 5057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:40.405280 5057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:40.409232 5057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:40.409286 5057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:40.409307 5057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:40.409327 5057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:40.409348 5057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:53:40.414458 5057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:42185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:42185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:40.415334 5057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:40.416245 5057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:40.424468 5069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:40.424443 5063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:40.424417 5064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:40.424502 5057 server_base.cc:1034] running on GCE node
I20260809 07:53:40.425156 5057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:40.426251 5057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:40.427428 5057 hybrid_clock.cc:648] HybridClock initialized: now 1786262020427407 us; error 31 us; skew 500 ppm
Aug 09 07:53:40 dist-test-slave-vpvm krb5kdc[5033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262020, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:40.433226 5057 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:53:40.434156 5057 webserver.cc:460] Webserver started at http://127.28.112.62:38043/ using document root <none> and password file <none>
I20260809 07:53:40.434667 5057 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:40.434728 5057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:40.434901 5057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:40.436833 5057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "dc0bbfc33c3141cabc78cf13bc484ab5"
format_stamp: "Formatted at 2026-08-09 07:53:40 on dist-test-slave-vpvm"
server_key: "6c7130b324cb19f5de0d74024ec7b028"
server_key_iv: "7946e1649d0f7c28078d643a39e58171"
server_key_version: "encryptionkey@0"
I20260809 07:53:40.437359 5057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "dc0bbfc33c3141cabc78cf13bc484ab5"
format_stamp: "Formatted at 2026-08-09 07:53:40 on dist-test-slave-vpvm"
server_key: "6c7130b324cb19f5de0d74024ec7b028"
server_key_iv: "7946e1649d0f7c28078d643a39e58171"
server_key_version: "encryptionkey@0"
I20260809 07:53:40.441071 5057 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:53:40.443435 5082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:40.444562 5057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:40.444666 5057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "dc0bbfc33c3141cabc78cf13bc484ab5"
format_stamp: "Formatted at 2026-08-09 07:53:40 on dist-test-slave-vpvm"
server_key: "6c7130b324cb19f5de0d74024ec7b028"
server_key_iv: "7946e1649d0f7c28078d643a39e58171"
server_key_version: "encryptionkey@0"
I20260809 07:53:40.444761 5057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:40.458230 5057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:40.460665 5057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:40.460815 5057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:40.467367 5057 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:42185
I20260809 07:53:40.467392 5155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:42185 every 8 connection(s)
I20260809 07:53:40.468197 5057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:53:40.470140 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 5057
I20260809 07:53:40.470243 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:53:40.470494 29120 external_mini_cluster.cc:1428] Setting key 465b1a990ee133dff4275e2864ed9a02
I20260809 07:53:40.470970 5158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:40.477631 5158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5: Bootstrap starting.
Aug 09 07:53:40 dist-test-slave-vpvm krb5kdc[5033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:40.479879 5158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:40.480738 5158 log.cc:824] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:40.482862 5158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5: No bootstrap required, opened a new log
I20260809 07:53:40.484778 5164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:40.471795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.28.112.62:42185)
0809 07:53:40.472216 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:40.472235 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:40.472278 (+ 43us) server_negotiation.cc:407] Connection header received
0809 07:53:40.473026 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:40.473050 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:40.473461 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:40.473848 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:40.474541 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:40.475950 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:40.476835 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:40.477492 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:40.477653 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:40.479954 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:40.479960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:40.479976 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:40.480009 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:40.481763 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:40.482204 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:40.482210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:40.482216 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:40.482357 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:40.482622 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:40.482625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:40.482626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:40.483112 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:40.483329 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:40.483595 (+ 266us) server_negotiation.cc:299] Negotiation successful
0809 07:53:40.483878 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":127,"threads_started":1}
I20260809 07:53:40.487046 5158 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc0bbfc33c3141cabc78cf13bc484ab5" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 42185 } }
I20260809 07:53:40.487259 5158 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:40.487301 5158 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc0bbfc33c3141cabc78cf13bc484ab5, State: Initialized, Role: FOLLOWER
I20260809 07:53:40.487650 5158 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc0bbfc33c3141cabc78cf13bc484ab5" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 42185 } }
I20260809 07:53:40.487757 5158 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:53:40.487800 5158 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:53:40.487865 5158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:40.488603 5158 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc0bbfc33c3141cabc78cf13bc484ab5" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 42185 } }
I20260809 07:53:40.488900 5158 leader_election.cc:302] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc0bbfc33c3141cabc78cf13bc484ab5; no voters:
I20260809 07:53:40.489131 5158 leader_election.cc:288] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:53:40.489271 5166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:40.489450 5166 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5 [term 1 LEADER]: Becoming Leader. State: Replica: dc0bbfc33c3141cabc78cf13bc484ab5, State: Running, Role: LEADER
I20260809 07:53:40.489797 5166 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc0bbfc33c3141cabc78cf13bc484ab5" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 42185 } }
I20260809 07:53:40.490346 5158 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:40.491420 5167 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc0bbfc33c3141cabc78cf13bc484ab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc0bbfc33c3141cabc78cf13bc484ab5" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 42185 } } }
I20260809 07:53:40.491428 5168 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc0bbfc33c3141cabc78cf13bc484ab5. Latest consensus state: current_term: 1 leader_uuid: "dc0bbfc33c3141cabc78cf13bc484ab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc0bbfc33c3141cabc78cf13bc484ab5" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 42185 } } }
I20260809 07:53:40.491569 5168 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:40.491567 5167 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:40.492005 5179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:53:40.494717 5179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:40.499955 5179 catalog_manager.cc:1347] Generated new cluster ID: aa24563d6b574150a99a1ecbf32728a7
I20260809 07:53:40.500017 5179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:40.537278 5179 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:53:40.538589 5179 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:40.552107 5179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc0bbfc33c3141cabc78cf13bc484ab5: Generated new TSK 0
I20260809 07:53:40.552809 5179 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
I20260809 07:53:40.586874 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:42185
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:40.697929 5198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:40.698197 5198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:40.698237 5198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:40.702121 5198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:40.702167 5198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:40.702219 5198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:53:40.707162 5198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:42185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:40.708058 5198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:40.708952 5198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:40.717770 5204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:40.717804 5209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:40.717772 5207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:40.717806 5198 server_base.cc:1034] running on GCE node
I20260809 07:53:40.718369 5198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:40.718930 5198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:40.720091 5198 hybrid_clock.cc:648] HybridClock initialized: now 1786262020720070 us; error 35 us; skew 500 ppm
Aug 09 07:53:40 dist-test-slave-vpvm krb5kdc[5033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262020, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:40.722596 5198 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:53:40.723398 5198 webserver.cc:460] Webserver started at http://127.28.112.1:35559/ using document root <none> and password file <none>
I20260809 07:53:40.723893 5198 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:40.723966 5198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:40.724154 5198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:40.726063 5198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "07c8ae66208a4ccda037508923b9f92b"
format_stamp: "Formatted at 2026-08-09 07:53:40 on dist-test-slave-vpvm"
server_key: "798aeec638a98129f19c993f1ab762e6"
server_key_iv: "fb8678b105a592b98c63c8e34a5cd933"
server_key_version: "encryptionkey@0"
I20260809 07:53:40.726560 5198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "07c8ae66208a4ccda037508923b9f92b"
format_stamp: "Formatted at 2026-08-09 07:53:40 on dist-test-slave-vpvm"
server_key: "798aeec638a98129f19c993f1ab762e6"
server_key_iv: "fb8678b105a592b98c63c8e34a5cd933"
server_key_version: "encryptionkey@0"
I20260809 07:53:40.729892 5198 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 07:53:40.732038 5220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:40.733117 5198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:40.733230 5198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "07c8ae66208a4ccda037508923b9f92b"
format_stamp: "Formatted at 2026-08-09 07:53:40 on dist-test-slave-vpvm"
server_key: "798aeec638a98129f19c993f1ab762e6"
server_key_iv: "fb8678b105a592b98c63c8e34a5cd933"
server_key_version: "encryptionkey@0"
I20260809 07:53:40.733314 5198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:40.754438 5198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:40.757149 5198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:40.757320 5198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:40.757889 5198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:40.758706 5198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:40.758754 5198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:40.758795 5198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:40.758826 5198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:40.767088 5198 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:42597
I20260809 07:53:40.767113 5388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:42597 every 8 connection(s)
I20260809 07:53:40.767910 5198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:53:40.772831 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 5198
I20260809 07:53:40.772946 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:53:40.773229 29120 external_mini_cluster.cc:1428] Setting key 53a0c4ec1283ab03dbb6b315309d48cc
Aug 09 07:53:40 dist-test-slave-vpvm krb5kdc[5033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262020, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:40.780790 5164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:40.769515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:43361 (local address 127.28.112.62:42185)
0809 07:53:40.769635 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:40.769641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:40.770445 (+ 804us) server_negotiation.cc:407] Connection header received
0809 07:53:40.771761 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:40.771765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:40.771825 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:40.771899 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:40.773082 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:40.773776 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:40.774972 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:40.775209 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:40.775298 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:40.777519 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:40.777522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:40.777524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:40.777547 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:40.778897 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:40.779364 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:40.779368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:40.779371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:40.779469 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:40.779762 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:40.779766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:40.779767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:40.779981 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:40.780083 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:40.780483 (+ 400us) server_negotiation.cc:299] Negotiation successful
0809 07:53:40.780616 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 07:53:40.781392 5393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:40.769852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:42185 (local address 127.28.112.1:43361)
0809 07:53:40.770303 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:40.770344 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:40.771553 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:40.772071 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:40.772078 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:40.772431 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:40.772909 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:40.772925 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:40.773897 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:40.773901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:40.774856 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:40.774867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:40.775345 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:40.775349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:40.775501 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:40.775995 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:40.776011 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:40.777366 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:40.779003 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:40.779008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:40.779018 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:40.779260 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:40.779553 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:40.779556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:40.779557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:40.779680 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:40.780067 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:40.780071 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:40.780268 (+ 197us) client_negotiation.cc:769] Sending connection context
0809 07:53:40.780407 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 07:53:40.780649 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":108,"threads_started":1}
I20260809 07:53:40.782536 5390 heartbeater.cc:347] Connected to a master server at 127.28.112.62:42185
I20260809 07:53:40.782750 5390 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:40.783214 5390 heartbeater.cc:511] Master 127.28.112.62:42185 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.112.2@KRBTEST.COM; defaulting to no policy
I20260809 07:53:40.784680 5106 ts_manager.cc:194] Registered new tserver with Master: 07c8ae66208a4ccda037508923b9f92b (127.28.112.1:42597)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.2@KRBTEST.COM" created.
I20260809 07:53:40.786051 5106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:43361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
I20260809 07:53:40.800312 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:42185
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:40.908680 5399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:40.908923 5399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:40.908979 5399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:40.912919 5399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:40.912976 5399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:40.913065 5399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:53:40.918277 5399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:42185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:40.919206 5399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:40.920171 5399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:40.929199 5414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:40.929255 5409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:40.929250 5406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:40.929412 5399 server_base.cc:1034] running on GCE node
I20260809 07:53:40.929831 5399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:40.930435 5399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:40.931586 5399 hybrid_clock.cc:648] HybridClock initialized: now 1786262020931564 us; error 34 us; skew 500 ppm
Aug 09 07:53:40 dist-test-slave-vpvm krb5kdc[5033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262020, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:40.934113 5399 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:53:40.934929 5399 webserver.cc:460] Webserver started at http://127.28.112.2:36547/ using document root <none> and password file <none>
I20260809 07:53:40.935446 5399 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:40.935511 5399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:40.935708 5399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:40.937660 5399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "c3eab4cdc3d048bfa7cb5e044394c0db"
format_stamp: "Formatted at 2026-08-09 07:53:40 on dist-test-slave-vpvm"
server_key: "ec372b4dec77a0b503aa5c9f3722f34d"
server_key_iv: "4034c4258515ac850ee7dd608b84a0b9"
server_key_version: "encryptionkey@0"
I20260809 07:53:40.938210 5399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "c3eab4cdc3d048bfa7cb5e044394c0db"
format_stamp: "Formatted at 2026-08-09 07:53:40 on dist-test-slave-vpvm"
server_key: "ec372b4dec77a0b503aa5c9f3722f34d"
server_key_iv: "4034c4258515ac850ee7dd608b84a0b9"
server_key_version: "encryptionkey@0"
I20260809 07:53:40.941583 5399 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 07:53:40.943902 5423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:40.945089 5399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:40.945195 5399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "c3eab4cdc3d048bfa7cb5e044394c0db"
format_stamp: "Formatted at 2026-08-09 07:53:40 on dist-test-slave-vpvm"
server_key: "ec372b4dec77a0b503aa5c9f3722f34d"
server_key_iv: "4034c4258515ac850ee7dd608b84a0b9"
server_key_version: "encryptionkey@0"
I20260809 07:53:40.945288 5399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:40.964416 5399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:40.966902 5399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:40.967062 5399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:40.967582 5399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:40.968343 5399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:40.968389 5399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:40.968427 5399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:40.968453 5399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:40.976771 5399 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:41839
I20260809 07:53:40.976794 5602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:41839 every 8 connection(s)
I20260809 07:53:40.977550 5399 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
Aug 09 07:53:40 dist-test-slave-vpvm krb5kdc[5033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262020, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:40.986215 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 5399
I20260809 07:53:40.986305 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:53:40.986574 29120 external_mini_cluster.cc:1428] Setting key c61d0167c65d8a9f298076b51d08d967
I20260809 07:53:40.990031 5164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:40.979174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:44093 (local address 127.28.112.62:42185)
0809 07:53:40.979294 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:40.979297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:40.980078 (+ 781us) server_negotiation.cc:407] Connection header received
0809 07:53:40.981321 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:40.981324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:40.981372 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:40.981441 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:40.982556 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:40.983073 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:40.984223 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:40.984441 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:40.984518 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:40.986781 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:40.986785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:40.986787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:40.986812 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:40.988078 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:40.988551 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:40.988554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:40.988555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:40.988632 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:40.988950 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:40.988953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:40.988955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:40.989127 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:40.989225 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:40.989720 (+ 495us) server_negotiation.cc:299] Negotiation successful
0809 07:53:40.989843 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 07:53:40.990841 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:40.979507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:42185 (local address 127.28.112.2:44093)
0809 07:53:40.979931 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:40.979970 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:40.981144 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:40.981635 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:40.981642 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:40.982007 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:40.982447 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:40.982460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:40.983218 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:40.983221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:40.984118 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:40.984128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:40.984550 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:40.984552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:40.984685 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:40.985147 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:40.985162 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:40.986574 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:40.988193 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:40.988199 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:40.988214 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:40.988454 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:40.988723 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:40.988726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:40.988728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:40.988861 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:40.989244 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:40.989251 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:40.989488 (+ 237us) client_negotiation.cc:769] Sending connection context
0809 07:53:40.989653 (+ 165us) client_negotiation.cc:240] Negotiation successful
0809 07:53:40.989977 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":108,"threads_started":1}
I20260809 07:53:40.991940 5603 heartbeater.cc:347] Connected to a master server at 127.28.112.62:42185
I20260809 07:53:40.992154 5603 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:40.992630 5603 heartbeater.cc:511] Master 127.28.112.62:42185 requested a full tablet report, sending...
I20260809 07:53:40.993716 5106 ts_manager.cc:194] Registered new tserver with Master: c3eab4cdc3d048bfa7cb5e044394c0db (127.28.112.2:41839)
I20260809 07:53:40.994504 5106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:44093
WARNING: no policy specified for kudu/127.28.112.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
I20260809 07:53:41.012192 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:42185
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:41.121320 5608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:41.121548 5608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:41.121590 5608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:41.125619 5608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:41.125684 5608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:41.125761 5608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:53:41.130913 5608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:42185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:41.131808 5608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:41.132714 5608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:41.141602 5617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:41.141731 5618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:41.141831 5608 server_base.cc:1034] running on GCE node
W20260809 07:53:41.141602 5623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:41.142288 5608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:41.142849 5608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:41.143998 5608 hybrid_clock.cc:648] HybridClock initialized: now 1786262021143978 us; error 31 us; skew 500 ppm
Aug 09 07:53:41 dist-test-slave-vpvm krb5kdc[5033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:41.146656 5608 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:53:41.147467 5608 webserver.cc:460] Webserver started at http://127.28.112.3:42155/ using document root <none> and password file <none>
I20260809 07:53:41.147969 5608 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:41.148026 5608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:41.148263 5608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:41.150358 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "cb4701ba92724f3c80821a0735a19a57"
format_stamp: "Formatted at 2026-08-09 07:53:41 on dist-test-slave-vpvm"
server_key: "88521a899132b4b44387af004791b8e3"
server_key_iv: "5579223e9d1003c2bbf7d742f12ccdf0"
server_key_version: "encryptionkey@0"
I20260809 07:53:41.150857 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "cb4701ba92724f3c80821a0735a19a57"
format_stamp: "Formatted at 2026-08-09 07:53:41 on dist-test-slave-vpvm"
server_key: "88521a899132b4b44387af004791b8e3"
server_key_iv: "5579223e9d1003c2bbf7d742f12ccdf0"
server_key_version: "encryptionkey@0"
I20260809 07:53:41.154304 5608 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 07:53:41.156608 5632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:41.157724 5608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:41.157821 5608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "cb4701ba92724f3c80821a0735a19a57"
format_stamp: "Formatted at 2026-08-09 07:53:41 on dist-test-slave-vpvm"
server_key: "88521a899132b4b44387af004791b8e3"
server_key_iv: "5579223e9d1003c2bbf7d742f12ccdf0"
server_key_version: "encryptionkey@0"
I20260809 07:53:41.157918 5608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:41.172531 5608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:41.175334 5608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:41.175505 5608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:41.176030 5608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:41.176823 5608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:41.176869 5608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:41.176911 5608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:41.176941 5608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:41.185261 5608 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:46053
I20260809 07:53:41.185308 5797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:46053 every 8 connection(s)
I20260809 07:53:41.186127 5608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:53:41.188019 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 5608
I20260809 07:53:41.188141 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:53:41.188413 29120 external_mini_cluster.cc:1428] Setting key a27830a3bb189e9e69ad852a6dbb92c9
Aug 09 07:53:41 dist-test-slave-vpvm krb5kdc[5033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:41.198760 5164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:41.187866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:48857 (local address 127.28.112.62:42185)
0809 07:53:41.188017 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:41.188023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:41.189025 (+ 1002us) server_negotiation.cc:407] Connection header received
0809 07:53:41.190225 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:41.190228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:41.190277 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:41.190343 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:41.191455 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:41.192041 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:41.193113 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:41.193317 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:41.193384 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:41.195600 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:41.195603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:41.195605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:41.195629 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:41.196839 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:41.197340 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:41.197343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:41.197344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:41.197419 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:41.197726 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:41.197729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:41.197731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:41.197901 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:41.198012 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:41.198474 (+ 462us) server_negotiation.cc:299] Negotiation successful
0809 07:53:41.198605 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 07:53:41.199465 5800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:41.188250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:42185 (local address 127.28.112.3:48857)
0809 07:53:41.188707 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:41.188757 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:41.190041 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:41.190517 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:41.190526 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:41.190891 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:41.191340 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:41.191354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:41.192152 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:41.192154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:41.193020 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:41.193029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:41.193417 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:41.193440 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:41.193570 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:41.194060 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:41.194077 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:41.195412 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:41.196961 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:41.196967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:41.196995 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:41.197244 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:41.197501 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:41.197504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:41.197506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:41.197637 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:41.198010 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:41.198016 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:41.198255 (+ 239us) client_negotiation.cc:769] Sending connection context
0809 07:53:41.198414 (+ 159us) client_negotiation.cc:240] Negotiation successful
0809 07:53:41.198716 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260809 07:53:41.200451 5798 heartbeater.cc:347] Connected to a master server at 127.28.112.62:42185
I20260809 07:53:41.200666 5798 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:41.201176 5798 heartbeater.cc:511] Master 127.28.112.62:42185 requested a full tablet report, sending...
I20260809 07:53:41.202268 5105 ts_manager.cc:194] Registered new tserver with Master: cb4701ba92724f3c80821a0735a19a57 (127.28.112.3:46053)
I20260809 07:53:41.202826 5105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:48857
I20260809 07:53:41.213856 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/test-admin.keytab.
Aug 09 07:53:41 dist-test-slave-vpvm krb5kdc[5033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:41.229779 29120 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260809 07:53:41.232287 5809 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:41.231007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35564 (local address 127.28.112.1:42597)
0809 07:53:41.231337 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:41.231347 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:41.231368 (+ 21us) server_negotiation.cc:407] Connection header received
0809 07:53:41.231414 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:41.231419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:41.231614 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:41.231725 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:41.232109 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35564: BlockingRecv error: recv got EOF from 127.0.0.1:35564 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":124,"threads_started":1}
W20260809 07:53:41.232417 29120 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/krb5cc' of type FILE
Aug 09 07:53:41 dist-test-slave-vpvm krb5kdc[5033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:41.233363 29120 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 07:53:41 dist-test-slave-vpvm krb5kdc[5033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:41.234252 29120 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 07:53:41 dist-test-slave-vpvm krb5kdc[5033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.1@KRBTEST.COM
I20260809 07:53:41.243188 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:41.235010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35574 (local address 127.28.112.1:42597)
0809 07:53:41.235139 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:41.235143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:41.235221 (+ 78us) server_negotiation.cc:407] Connection header received
0809 07:53:41.235370 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:41.235373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:41.235447 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:41.235516 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:41.235987 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:41.236772 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:41.237663 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:41.238092 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:41.238189 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:41.240179 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:41.240187 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:41.240199 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:41.240221 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:41.241669 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:41.242053 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:41.242059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:41.242061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:41.242150 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:41.242370 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:41.242372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:41.242374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:41.242614 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:41.242760 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:41.242889 (+ 129us) server_negotiation.cc:299] Negotiation successful
0809 07:53:41.243025 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260809 07:53:41.246459 5809 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:41.245582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.28.112.1:42597)
0809 07:53:41.245686 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:41.245689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:41.245745 (+ 56us) server_negotiation.cc:407] Connection header received
0809 07:53:41.245863 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:41.245866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:41.245900 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:41.246026 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:41.246379 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35576: BlockingRecv error: recv got EOF from 127.0.0.1:35576 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260809 07:53:41.246567 29120 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/krb5cc' of type FILE
Aug 09 07:53:41 dist-test-slave-vpvm krb5kdc[5033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:41.247413 29120 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 07:53:41 dist-test-slave-vpvm krb5kdc[5033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:41.248283 29120 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 07:53:41 dist-test-slave-vpvm krb5kdc[5033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.1@KRBTEST.COM
I20260809 07:53:41.255457 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:41.249085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35588 (local address 127.28.112.1:42597)
0809 07:53:41.249186 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:41.249190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:41.249313 (+ 123us) server_negotiation.cc:407] Connection header received
0809 07:53:41.249401 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:41.249403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:41.249434 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:41.249493 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:41.249922 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:41.250436 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:41.251116 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:41.251319 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:41.251380 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:41.253225 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:41.253228 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:41.253230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:41.253251 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:41.254326 (+ 1075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:41.254670 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:41.254672 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:41.254673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:41.254735 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:41.254906 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:41.254908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:41.254909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:41.255043 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:41.255127 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:41.255275 (+ 148us) server_negotiation.cc:299] Negotiation successful
0809 07:53:41.255356 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260809 07:53:41.257642 5809 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:41.256915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35602 (local address 127.28.112.1:42597)
0809 07:53:41.257017 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:41.257021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:41.257104 (+ 83us) server_negotiation.cc:407] Connection header received
0809 07:53:41.257189 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:41.257191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:41.257222 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:41.257281 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:41.257555 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35602: BlockingRecv error: recv got EOF from 127.0.0.1:35602 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260809 07:53:41.257871 29120 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786261991120089-29120-0/krb5kdc/krb5cc' of type FILE
Aug 09 07:53:41 dist-test-slave-vpvm krb5kdc[5033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:41.258746 29120 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 07:53:41 dist-test-slave-vpvm krb5kdc[5033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:41.259469 29120 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 07:53:41 dist-test-slave-vpvm krb5kdc[5033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.1@KRBTEST.COM
I20260809 07:53:41.266601 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:41.260166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35610 (local address 127.28.112.1:42597)
0809 07:53:41.260262 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:41.260265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:41.260377 (+ 112us) server_negotiation.cc:407] Connection header received
0809 07:53:41.260456 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:41.260458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:41.260488 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:41.260547 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:41.260935 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:41.261388 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:41.262195 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:41.262397 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:41.262474 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:41.264373 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:41.264376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:41.264377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:41.264398 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:41.265401 (+ 1003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:41.265753 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:41.265755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:41.265756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:41.265819 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:41.266021 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:41.266023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:41.266024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:41.266167 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:41.266255 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:41.266421 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 07:53:41.266503 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 07:53:41.267468 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 5198
I20260809 07:53:41.280790 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 5399
I20260809 07:53:41.307569 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 5608
I20260809 07:53:41.327221 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 5057
2026-08-09T07:53:41Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2281 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:53:41 dist-test-slave-vpvm krb5kdc[5823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:53:41 dist-test-slave-vpvm krb5kdc[5823](info): set up 2 sockets
Aug 09 07:53:41 dist-test-slave-vpvm krb5kdc[5823](info): commencing operation
krb5kdc: starting...
W20260809 07:53:42.462157 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:53:42 dist-test-slave-vpvm krb5kdc[5823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:53:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:53:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.62.keytab.
Entry for principal oryx/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.62.keytab.
I20260809 07:53:42.580386 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:37291
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.62.keytab
--principal=oryx/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:42033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:37291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:42.690719 5842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:42.690955 5842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:42.691008 5842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:42.694877 5842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:42.694926 5842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:42.694952 5842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:42.694975 5842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:42.695004 5842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:53:42.700078 5842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:42033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:37291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.62.keytab
--principal=oryx/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:37291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:42.701007 5842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:42.701921 5842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:42.709975 5854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:42.710047 5851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:42.710014 5850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:42.710196 5842 server_base.cc:1034] running on GCE node
I20260809 07:53:42.710795 5842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:42.711777 5842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:42.712941 5842 hybrid_clock.cc:648] HybridClock initialized: now 1786262022712925 us; error 25 us; skew 500 ppm
Aug 09 07:53:42 dist-test-slave-vpvm krb5kdc[5823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262022, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:42.715497 5842 init.cc:377] Logged in from keytab as oryx/127.28.112.62@KRBTEST.COM (short username oryx)
I20260809 07:53:42.716313 5842 webserver.cc:460] Webserver started at http://127.28.112.62:37879/ using document root <none> and password file <none>
I20260809 07:53:42.716813 5842 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:42.716874 5842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:42.717072 5842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:42.718974 5842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "64801c974cf64a69844c72bf20ac5983"
format_stamp: "Formatted at 2026-08-09 07:53:42 on dist-test-slave-vpvm"
server_key: "b509c17d4f28e8e11d0d91f8977d0d40"
server_key_iv: "a59c94633594c5c8c91e49102d0ca9fb"
server_key_version: "encryptionkey@0"
I20260809 07:53:42.719477 5842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "64801c974cf64a69844c72bf20ac5983"
format_stamp: "Formatted at 2026-08-09 07:53:42 on dist-test-slave-vpvm"
server_key: "b509c17d4f28e8e11d0d91f8977d0d40"
server_key_iv: "a59c94633594c5c8c91e49102d0ca9fb"
server_key_version: "encryptionkey@0"
I20260809 07:53:42.722936 5842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:53:42.725147 5865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:42.726351 5842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:42.726466 5842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "64801c974cf64a69844c72bf20ac5983"
format_stamp: "Formatted at 2026-08-09 07:53:42 on dist-test-slave-vpvm"
server_key: "b509c17d4f28e8e11d0d91f8977d0d40"
server_key_iv: "a59c94633594c5c8c91e49102d0ca9fb"
server_key_version: "encryptionkey@0"
I20260809 07:53:42.726550 5842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:42.736264 5842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:42.739023 5842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:42.739184 5842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:42.745983 5842 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:37291
I20260809 07:53:42.746238 5949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:37291 every 8 connection(s)
I20260809 07:53:42.746896 5842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:53:42.749495 5951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:42.754810 5951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983: Bootstrap starting.
I20260809 07:53:42.756350 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 5842
I20260809 07:53:42.756451 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:53:42.756697 29120 external_mini_cluster.cc:1428] Setting key 9f23eb576502c2cb3727bbd2bd57276a
I20260809 07:53:42.757277 5951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:42.758101 5951 log.cc:824] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:42.760167 5951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983: No bootstrap required, opened a new log
I20260809 07:53:42.764142 5951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } }
I20260809 07:53:42.764335 5951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:42.764370 5951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64801c974cf64a69844c72bf20ac5983, State: Initialized, Role: FOLLOWER
I20260809 07:53:42.764760 5951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } }
I20260809 07:53:42.764873 5951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:53:42.764941 5951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:53:42.765038 5951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [term 0 FOLLOWER]: Advancing to term 1
Aug 09 07:53:42 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM
I20260809 07:53:42.766004 5951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } }
I20260809 07:53:42.766331 5951 leader_election.cc:302] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64801c974cf64a69844c72bf20ac5983; no voters:
I20260809 07:53:42.766642 5951 leader_election.cc:288] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:53:42.766742 5957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:42.766963 5957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [term 1 LEADER]: Becoming Leader. State: Replica: 64801c974cf64a69844c72bf20ac5983, State: Running, Role: LEADER
I20260809 07:53:42.767328 5957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } }
I20260809 07:53:42.767890 5951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:42.769326 5964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64801c974cf64a69844c72bf20ac5983. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } }
I20260809 07:53:42.769359 5962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } }
I20260809 07:53:42.769858 5964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:42.769882 5962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:42.770390 5982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:53:42.772663 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:42.757766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.28.112.62:37291)
0809 07:53:42.758255 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:42.758273 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:42.758315 (+ 42us) server_negotiation.cc:407] Connection header received
0809 07:53:42.759045 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:42.759070 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:42.759460 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:42.759809 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:42.760477 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:42.761932 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:42.762763 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:42.763500 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:42.763662 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:42.766203 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:42.766209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:42.766226 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:42.766256 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:42.768162 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:42.769556 (+ 1394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:42.769562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:42.769567 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:42.769720 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:42.770077 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:42.770081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:42.770083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:42.770597 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:42.771350 (+ 753us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:42.771509 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 07:53:42.771789 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":173,"threads_started":1}
I20260809 07:53:42.773527 5982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:42.779135 5982 catalog_manager.cc:1347] Generated new cluster ID: 4b926155880248228e380cab0d664580
I20260809 07:53:42.779196 5982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:42.793996 5982 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:53:42.795125 5982 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:42.809644 5982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983: Generated new TSK 0
I20260809 07:53:42.810408 5982 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.1.keytab.
Entry for principal oryx/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.1.keytab.
I20260809 07:53:42.836840 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:37291
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.1.keytab
--principal=oryx/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:42033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:42.947862 5990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:42.948107 5990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:42.948146 5990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:42.952060 5990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:42.952107 5990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:42.952167 5990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:53:42.957118 5990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:42033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.1.keytab
--principal=oryx/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:37291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:42.958045 5990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:42.958977 5990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:42.967906 6005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:42.967916 6001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:42.968003 5990 server_base.cc:1034] running on GCE node
W20260809 07:53:42.967917 6002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:42.968561 5990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:42.969133 5990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:42.970350 5990 hybrid_clock.cc:648] HybridClock initialized: now 1786262022970321 us; error 38 us; skew 500 ppm
Aug 09 07:53:42 dist-test-slave-vpvm krb5kdc[5823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262022, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:42.972980 5990 init.cc:377] Logged in from keytab as oryx/127.28.112.1@KRBTEST.COM (short username oryx)
I20260809 07:53:42.973765 5990 webserver.cc:460] Webserver started at http://127.28.112.1:45283/ using document root <none> and password file <none>
I20260809 07:53:42.974315 5990 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:42.974376 5990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:42.974570 5990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:42.976511 5990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "730d23ec728547609406d0fceaf73197"
format_stamp: "Formatted at 2026-08-09 07:53:42 on dist-test-slave-vpvm"
server_key: "c4e7ea4086348b68398c3bdeaad1f108"
server_key_iv: "3b1ffe4e188887d4ccdaa73f67531b5e"
server_key_version: "encryptionkey@0"
I20260809 07:53:42.977028 5990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "730d23ec728547609406d0fceaf73197"
format_stamp: "Formatted at 2026-08-09 07:53:42 on dist-test-slave-vpvm"
server_key: "c4e7ea4086348b68398c3bdeaad1f108"
server_key_iv: "3b1ffe4e188887d4ccdaa73f67531b5e"
server_key_version: "encryptionkey@0"
I20260809 07:53:42.980602 5990 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 07:53:42.982884 6014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:42.984043 5990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:42.984144 5990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "730d23ec728547609406d0fceaf73197"
format_stamp: "Formatted at 2026-08-09 07:53:42 on dist-test-slave-vpvm"
server_key: "c4e7ea4086348b68398c3bdeaad1f108"
server_key_iv: "3b1ffe4e188887d4ccdaa73f67531b5e"
server_key_version: "encryptionkey@0"
I20260809 07:53:42.984242 5990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:43.003722 5990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:43.006508 5990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:43.006680 5990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:43.007262 5990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:43.008082 5990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:43.008128 5990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:43.008172 5990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:43.008200 5990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:43.016803 5990 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:43077
I20260809 07:53:43.016834 6182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:43077 every 8 connection(s)
I20260809 07:53:43.017638 5990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:53:43.022698 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 5990
I20260809 07:53:43.022825 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:53:43.023114 29120 external_mini_cluster.cc:1428] Setting key eecdc06aac1ea14213a611f480fbdb22
Aug 09 07:53:43 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262022, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM
I20260809 07:53:43.031689 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.019352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:47015 (local address 127.28.112.62:37291)
0809 07:53:43.019516 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:43.019520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:43.020300 (+ 780us) server_negotiation.cc:407] Connection header received
0809 07:53:43.021594 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:43.021597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:43.021645 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:43.021717 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:43.022974 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.023629 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.024962 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.025205 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.025294 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.027881 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:43.027886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:43.027888 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:43.027928 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:43.029583 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.030083 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.030087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.030089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.030194 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.030532 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.030536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.030538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.030781 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:43.030905 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:43.031353 (+ 448us) server_negotiation.cc:299] Negotiation successful
0809 07:53:43.031492 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 07:53:43.032308 6188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.019710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.1:47015)
0809 07:53:43.020158 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:43.020199 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:43.021399 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:43.021859 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:43.021866 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:43.022226 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:43.022817 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.022839 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.023761 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.023765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.024843 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.024854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.025356 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.025360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.025545 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.026178 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:43.026198 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:43.027713 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:43.029705 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.029709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.029721 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.029989 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.030292 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.030295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.030296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.030427 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.030893 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:43.030899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:43.031111 (+ 212us) client_negotiation.cc:769] Sending connection context
0809 07:53:43.031256 (+ 145us) client_negotiation.cc:240] Negotiation successful
0809 07:53:43.031487 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":123,"threads_started":1}
I20260809 07:53:43.033687 6184 heartbeater.cc:347] Connected to a master server at 127.28.112.62:37291
WARNING: no policy specified for oryx/127.28.112.2@KRBTEST.COM; defaulting to no policy
I20260809 07:53:43.033907 6184 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:43.034379 6184 heartbeater.cc:511] Master 127.28.112.62:37291 requested a full tablet report, sending...
I20260809 07:53:43.035848 5902 ts_manager.cc:194] Registered new tserver with Master: 730d23ec728547609406d0fceaf73197 (127.28.112.1:43077)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.112.2@KRBTEST.COM" created.
I20260809 07:53:43.037120 5902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:47015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.2.keytab.
Entry for principal oryx/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.2.keytab.
I20260809 07:53:43.050048 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:37291
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.2.keytab
--principal=oryx/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:42033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:43.157353 6191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:43.157605 6191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:43.157650 6191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:43.161540 6191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:43.161593 6191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:43.161661 6191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:53:43.166721 6191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:42033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.2.keytab
--principal=oryx/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:37291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:43.167680 6191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:43.168678 6191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:43.177776 6200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:43.177786 6203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:43.177867 6201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:43.178117 6191 server_base.cc:1034] running on GCE node
I20260809 07:53:43.178505 6191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:43.179126 6191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:43.180301 6191 hybrid_clock.cc:648] HybridClock initialized: now 1786262023180281 us; error 30 us; skew 500 ppm
Aug 09 07:53:43 dist-test-slave-vpvm krb5kdc[5823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262023, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:43.182930 6191 init.cc:377] Logged in from keytab as oryx/127.28.112.2@KRBTEST.COM (short username oryx)
I20260809 07:53:43.183740 6191 webserver.cc:460] Webserver started at http://127.28.112.2:34257/ using document root <none> and password file <none>
I20260809 07:53:43.184243 6191 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:43.184299 6191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:43.184458 6191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:43.186414 6191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0"
format_stamp: "Formatted at 2026-08-09 07:53:43 on dist-test-slave-vpvm"
server_key: "9c92d5167e9b26bb5abd9ec49c170685"
server_key_iv: "b6a5d6ae45642af676e52573e1358179"
server_key_version: "encryptionkey@0"
I20260809 07:53:43.186926 6191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0"
format_stamp: "Formatted at 2026-08-09 07:53:43 on dist-test-slave-vpvm"
server_key: "9c92d5167e9b26bb5abd9ec49c170685"
server_key_iv: "b6a5d6ae45642af676e52573e1358179"
server_key_version: "encryptionkey@0"
I20260809 07:53:43.190428 6191 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:53:43.192600 6213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:43.193693 6191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:43.193789 6191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0"
format_stamp: "Formatted at 2026-08-09 07:53:43 on dist-test-slave-vpvm"
server_key: "9c92d5167e9b26bb5abd9ec49c170685"
server_key_iv: "b6a5d6ae45642af676e52573e1358179"
server_key_version: "encryptionkey@0"
I20260809 07:53:43.193863 6191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:43.211627 6191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:43.214497 6191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:43.214638 6191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:43.215129 6191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:43.215890 6191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:43.215938 6191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:43.215975 6191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:43.215996 6191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:43.223992 6191 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:39549
I20260809 07:53:43.224040 6387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:39549 every 8 connection(s)
I20260809 07:53:43.224810 6191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:53:43.225415 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 6191
I20260809 07:53:43.225581 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:53:43.225885 29120 external_mini_cluster.cc:1428] Setting key b6b8ff3c54b10c917097b4eeb63d2caf
Aug 09 07:53:43 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262023, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.28.112.3@KRBTEST.COM; defaulting to no policy
I20260809 07:53:43.237900 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.226730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:56557 (local address 127.28.112.62:37291)
0809 07:53:43.226856 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:43.226860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:43.227686 (+ 826us) server_negotiation.cc:407] Connection header received
0809 07:53:43.228997 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:43.229001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:43.229054 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:43.229123 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:43.230248 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.230910 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.232067 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.232399 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.232508 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.234840 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:43.234843 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:43.234845 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:43.234868 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:43.236218 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.236707 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.236710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.236711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.236783 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.237048 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.237050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.237051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.237203 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:43.237271 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:43.237674 (+ 403us) server_negotiation.cc:299] Negotiation successful
0809 07:53:43.237783 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.112.3@KRBTEST.COM" created.
I20260809 07:53:43.238759 6390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.227046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.2:56557)
0809 07:53:43.227553 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:43.227592 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:43.228796 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:43.229294 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:43.229302 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:43.229646 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:43.230105 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.230120 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.231043 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.231046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.231920 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.231933 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.232529 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.232531 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.232673 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.233167 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:43.233183 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:43.234676 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:43.236328 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.236333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.236344 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.236626 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.236849 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.236851 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.236852 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.236980 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.237272 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:43.237277 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:43.237479 (+ 202us) client_negotiation.cc:769] Sending connection context
0809 07:53:43.237634 (+ 155us) client_negotiation.cc:240] Negotiation successful
0809 07:53:43.237887 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":136,"threads_started":1}
I20260809 07:53:43.239874 6388 heartbeater.cc:347] Connected to a master server at 127.28.112.62:37291
I20260809 07:53:43.240098 6388 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:43.240631 6388 heartbeater.cc:511] Master 127.28.112.62:37291 requested a full tablet report, sending...
I20260809 07:53:43.241730 5902 ts_manager.cc:194] Registered new tserver with Master: 94d6943ad70f4d11a5e4f6f0a1244bc0 (127.28.112.2:39549)
I20260809 07:53:43.242305 5902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:56557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.3.keytab.
Entry for principal oryx/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.3.keytab.
I20260809 07:53:43.252496 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:37291
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.3.keytab
--principal=oryx/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:42033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:43.360229 6393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:43.360479 6393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:43.360539 6393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:43.364437 6393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:43.364490 6393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:43.364567 6393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:53:43.369660 6393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:42033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.3.keytab
--principal=oryx/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:37291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:43.370612 6393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:43.371529 6393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:43.380604 6403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:43.380590 6405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:43.380759 6393 server_base.cc:1034] running on GCE node
W20260809 07:53:43.380931 6402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:43.381325 6393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:43.381923 6393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:43.383107 6393 hybrid_clock.cc:648] HybridClock initialized: now 1786262023383093 us; error 30 us; skew 500 ppm
Aug 09 07:53:43 dist-test-slave-vpvm krb5kdc[5823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262023, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:43.385695 6393 init.cc:377] Logged in from keytab as oryx/127.28.112.3@KRBTEST.COM (short username oryx)
I20260809 07:53:43.386545 6393 webserver.cc:460] Webserver started at http://127.28.112.3:34869/ using document root <none> and password file <none>
I20260809 07:53:43.387099 6393 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:43.387176 6393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:43.387408 6393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:43.389381 6393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "433a721039ab44cdbf224fda5eacdfdc"
format_stamp: "Formatted at 2026-08-09 07:53:43 on dist-test-slave-vpvm"
server_key: "ac9f77f9eb758e04eecb4bcdabe19167"
server_key_iv: "9f2847e28f88dda0132131afd49a5fa6"
server_key_version: "encryptionkey@0"
I20260809 07:53:43.389878 6393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "433a721039ab44cdbf224fda5eacdfdc"
format_stamp: "Formatted at 2026-08-09 07:53:43 on dist-test-slave-vpvm"
server_key: "ac9f77f9eb758e04eecb4bcdabe19167"
server_key_iv: "9f2847e28f88dda0132131afd49a5fa6"
server_key_version: "encryptionkey@0"
I20260809 07:53:43.393530 6393 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 07:53:43.395821 6415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:43.396981 6393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:43.397101 6393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "433a721039ab44cdbf224fda5eacdfdc"
format_stamp: "Formatted at 2026-08-09 07:53:43 on dist-test-slave-vpvm"
server_key: "ac9f77f9eb758e04eecb4bcdabe19167"
server_key_iv: "9f2847e28f88dda0132131afd49a5fa6"
server_key_version: "encryptionkey@0"
I20260809 07:53:43.397250 6393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:43.424268 6393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:43.427412 6393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:43.427591 6393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:43.428206 6393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:43.429067 6393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:43.429112 6393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:43.429167 6393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:43.429194 6393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:43.437836 6393 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:44345
I20260809 07:53:43.437860 6586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:44345 every 8 connection(s)
I20260809 07:53:43.438723 6393 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
Aug 09 07:53:43 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262023, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM
I20260809 07:53:43.448099 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 6393
I20260809 07:53:43.448215 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:53:43.448546 29120 external_mini_cluster.cc:1428] Setting key 86b55dd3c15fa42ec4e161e781cbbb4d
I20260809 07:53:43.451798 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.440463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:48123 (local address 127.28.112.62:37291)
0809 07:53:43.440603 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:43.440606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:43.441401 (+ 795us) server_negotiation.cc:407] Connection header received
0809 07:53:43.442811 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:43.442815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:43.442874 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:43.442948 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:43.444124 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.444656 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.445759 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.446005 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.446096 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.448514 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:43.448517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:43.448519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:43.448545 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:43.450039 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.450498 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.450502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.450504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.450599 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.450937 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.450940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.450941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.451108 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:43.451192 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:43.451589 (+ 397us) server_negotiation.cc:299] Negotiation successful
0809 07:53:43.451682 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 07:53:43.452536 6589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.440815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.3:48123)
0809 07:53:43.441261 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:43.441302 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:43.442593 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:43.443130 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:43.443137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:43.443502 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:43.443979 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.443993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.444778 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.444781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.445631 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.445651 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.446145 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.446149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.446315 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.446829 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:43.446848 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:43.448351 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:43.450141 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.450146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.450157 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.450403 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.450684 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.450686 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.450687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.450858 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.451190 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:43.451195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:43.451408 (+ 213us) client_negotiation.cc:769] Sending connection context
0809 07:53:43.451558 (+ 150us) client_negotiation.cc:240] Negotiation successful
0809 07:53:43.451793 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":103,"threads_started":1}
I20260809 07:53:43.453555 6587 heartbeater.cc:347] Connected to a master server at 127.28.112.62:37291
I20260809 07:53:43.453754 6587 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:43.454211 6587 heartbeater.cc:511] Master 127.28.112.62:37291 requested a full tablet report, sending...
I20260809 07:53:43.455148 5902 ts_manager.cc:194] Registered new tserver with Master: 433a721039ab44cdbf224fda5eacdfdc (127.28.112.3:44345)
I20260809 07:53:43.455669 5902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:48123
I20260809 07:53:43.462235 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.112.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.112.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.112.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.61.keytab.
Entry for principal oryx/127.28.112.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.61.keytab.
I20260809 07:53:43.488250 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.61:42839
--webserver_interface=127.28.112.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.61.keytab
--principal=oryx/127.28.112.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:42033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:37291,127.28.112.61:42839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:43.599718 6592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:43.599944 6592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:43.600006 6592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:43.604035 6592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:43.604090 6592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:43.604110 6592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:43.604123 6592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:43.604139 6592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:53:43.610055 6592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:42033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:37291,127.28.112.61:42839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.61.keytab
--principal=oryx/127.28.112.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.61:42839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.112.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
W20260809 07:53:43.610247 6592 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.112.62:37291,127.28.112.61:42839'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260809 07:53:43.611049 6592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:43.612020 6592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:43.620045 6604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:43.620110 6601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:43.620088 6602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:43.620235 6592 server_base.cc:1034] running on GCE node
I20260809 07:53:43.620841 6592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:43.621834 6592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:43.623000 6592 hybrid_clock.cc:648] HybridClock initialized: now 1786262023622979 us; error 35 us; skew 500 ppm
Aug 09 07:53:43 dist-test-slave-vpvm krb5kdc[5823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262023, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:43.625522 6592 init.cc:377] Logged in from keytab as oryx/127.28.112.61@KRBTEST.COM (short username oryx)
I20260809 07:53:43.626358 6592 webserver.cc:460] Webserver started at http://127.28.112.61:40505/ using document root <none> and password file <none>
I20260809 07:53:43.626845 6592 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:43.626902 6592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:43.627063 6592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:43.628955 6592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/data/instance:
uuid: "bbbca9ad28064f5bab856505d7dff710"
format_stamp: "Formatted at 2026-08-09 07:53:43 on dist-test-slave-vpvm"
server_key: "d2f08c74a5eb08df8885dff80e49ad0f"
server_key_iv: "7f90b4c523094cbe01544cb2088e0479"
server_key_version: "encryptionkey@0"
I20260809 07:53:43.629467 6592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/wal/instance:
uuid: "bbbca9ad28064f5bab856505d7dff710"
format_stamp: "Formatted at 2026-08-09 07:53:43 on dist-test-slave-vpvm"
server_key: "d2f08c74a5eb08df8885dff80e49ad0f"
server_key_iv: "7f90b4c523094cbe01544cb2088e0479"
server_key_version: "encryptionkey@0"
I20260809 07:53:43.632949 6592 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:53:43.635181 6613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:43.636296 6592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:43.636392 6592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/wal
uuid: "bbbca9ad28064f5bab856505d7dff710"
format_stamp: "Formatted at 2026-08-09 07:53:43 on dist-test-slave-vpvm"
server_key: "d2f08c74a5eb08df8885dff80e49ad0f"
server_key_iv: "7f90b4c523094cbe01544cb2088e0479"
server_key_version: "encryptionkey@0"
I20260809 07:53:43.636476 6592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:43.657701 6592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:43.660149 6592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:43.660290 6592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:43.666616 6592 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.61:42839
I20260809 07:53:43.666877 6695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.61:42839 every 8 connection(s)
I20260809 07:53:43.667548 6592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/data/info.pb
I20260809 07:53:43.670351 6696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:43.673785 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 6592
I20260809 07:53:43.673995 6696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 07:53:43 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262023, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.61@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM
Aug 09 07:53:43 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.112.61@KRBTEST.COM
I20260809 07:53:43.687014 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.675807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49490 (local address 127.28.112.62:37291)
0809 07:53:43.675950 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:43.675955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:43.676256 (+ 301us) server_negotiation.cc:407] Connection header received
0809 07:53:43.676710 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:43.676713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:43.676762 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:43.676866 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:43.677847 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.678716 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.680105 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.680448 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.680554 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.683167 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:43.683171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:43.683174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:43.683205 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:43.684690 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.685352 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.685356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.685357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.685441 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.685755 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.685758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.685760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.686064 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:43.686165 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:43.686701 (+ 536us) server_negotiation.cc:299] Negotiation successful
0809 07:53:43.686820 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 07:53:43.687976 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.675740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:49490)
0809 07:53:43.676123 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:43.676169 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:43.676528 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:43.676913 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:43.676918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:43.677357 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:43.677705 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.677725 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.678849 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.678852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.679986 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.679995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.680582 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.680585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.680712 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.681413 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:43.681433 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:43.682999 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:43.684801 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.684810 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.684820 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.685251 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.685514 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.685517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.685518 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.685647 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.686170 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:43.686177 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:43.686469 (+ 292us) client_negotiation.cc:769] Sending connection context
0809 07:53:43.686636 (+ 167us) client_negotiation.cc:240] Negotiation successful
0809 07:53:43.687031 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":1,"thread_start_us":138,"threads_started":1}
I20260809 07:53:43.687975 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.675401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.28.112.61:42839)
0809 07:53:43.675784 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:43.675797 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:43.675829 (+ 32us) server_negotiation.cc:407] Connection header received
0809 07:53:43.676521 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:43.676539 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:43.676749 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:43.676920 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:43.677774 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.678989 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.680099 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.680653 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.680732 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.683627 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:43.683632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:43.683652 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:43.683673 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:43.685255 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.685843 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.685848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.685849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.685917 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.686197 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.686200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.686201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.686475 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:43.686692 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:43.686846 (+ 154us) server_negotiation.cc:299] Negotiation successful
0809 07:53:43.687077 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":152,"server-negotiator.queue_time_us":245,"spinlock_wait_cycles":13952,"thread_start_us":95,"threads_started":1}
I20260809 07:53:43.689241 6696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 07:53:43 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262023, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.61@KRBTEST.COM for oryx/127.28.112.61@KRBTEST.COM
I20260809 07:53:43.696492 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.689777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49098 (local address 127.28.112.61:42839)
0809 07:53:43.690100 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:43.690103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:43.690112 (+ 9us) server_negotiation.cc:407] Connection header received
0809 07:53:43.690137 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:43.690139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:43.690170 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:43.690232 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:43.690578 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.691367 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.692068 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.692278 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.692360 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.694082 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:43.694086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:43.694089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:43.694111 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:43.695146 (+ 1035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.695497 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.695500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.695502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.695566 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.695795 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.695798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.695800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.695947 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:43.696029 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:43.696254 (+ 225us) server_negotiation.cc:299] Negotiation successful
0809 07:53:43.696375 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":268}
I20260809 07:53:43.696491 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.689727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:42839 (local address 127.0.0.1:49098)
0809 07:53:43.689821 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:43.689836 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:43.689903 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:43.690259 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:43.690262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:43.690403 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:43.690481 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.690485 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.691466 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.691469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.691969 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.691976 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.692367 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.692370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.692499 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.692840 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:43.692850 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:43.693977 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:43.695241 (+ 1264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.695244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.695246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.695416 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.695643 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.695646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.695648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.695722 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.696030 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:43.696033 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:43.696110 (+ 77us) client_negotiation.cc:769] Sending connection context
0809 07:53:43.696199 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 07:53:43.696327 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":30}
I20260809 07:53:43.699370 6696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: Bootstrap starting.
I20260809 07:53:43.701589 6696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:43.702332 6696 log.cc:824] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:43.704053 6696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: No bootstrap required, opened a new log
I20260809 07:53:43.706913 6696 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } }
I20260809 07:53:43.707108 6696 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:43.707147 6696 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbbca9ad28064f5bab856505d7dff710, State: Initialized, Role: FOLLOWER
I20260809 07:53:43.707532 6696 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } }
I20260809 07:53:43.707990 6704 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } } }
I20260809 07:53:43.708108 6704 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:53:43.708660 6696 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 07:53:43.713078 6722 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:53:43.713162 6722 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:53:43.714728 6592 master_runner.cc:428] Detected that this master bbbca9ad28064f5bab856505d7dff710 is joining an existing cluster
I20260809 07:53:43.714775 6592 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.112.61:42839
I20260809 07:53:43.724421 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.716724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.28.112.62:37291)
0809 07:53:43.716843 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:43.716847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:43.717074 (+ 227us) server_negotiation.cc:407] Connection header received
0809 07:53:43.717249 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:43.717253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:43.717301 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:43.717350 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:43.718050 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.718797 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.719604 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.719838 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.719933 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.721184 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:43.721187 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:43.721190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:43.721212 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:43.722791 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.723338 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.723342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.723344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.723429 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.723673 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.723676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.723678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.723852 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:43.723936 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:43.724187 (+ 251us) server_negotiation.cc:299] Negotiation successful
0809 07:53:43.724292 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 07:53:43.724412 6731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.716631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:49498)
0809 07:53:43.716996 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:43.717010 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:43.717141 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:43.717490 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:43.717493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:43.717781 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:43.717900 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.717907 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.718886 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.718889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.719498 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.719507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.719961 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.719964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.720129 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.720593 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:43.720609 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:43.721070 (+ 461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:43.722905 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.722908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.722910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.723233 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.723509 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.723512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.723514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.723590 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.723941 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:43.723944 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:43.724027 (+ 83us) client_negotiation.cc:769] Sending connection context
0809 07:53:43.724111 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 07:53:43.724236 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":58,"thread_start_us":63,"threads_started":1}
I20260809 07:53:43.724562 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.716667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49114 (local address 127.28.112.61:42839)
0809 07:53:43.716768 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:43.716772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:43.717203 (+ 431us) server_negotiation.cc:407] Connection header received
0809 07:53:43.717308 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:43.717311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:43.717346 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:43.717403 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:43.717871 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.718657 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.719813 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.720161 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.720247 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.721153 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:43.721157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:43.721160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:43.721183 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:43.722926 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.723364 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.723367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.723369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.723447 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.723717 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.723720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.723722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.723917 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:43.723997 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:43.724296 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 07:53:43.724394 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":39}
I20260809 07:53:43.724984 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.716636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:42839 (local address 127.0.0.1:49114)
0809 07:53:43.717121 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:43.717133 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:43.717226 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:43.717424 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:43.717427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:43.717662 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:43.717759 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.717775 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.718775 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.718779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.719709 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.719720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.720261 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.720265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.720348 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.720709 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:43.720719 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:43.721029 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:43.723012 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.723015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.723017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.723261 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.723541 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.723544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.723546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.723626 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.724056 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:43.724059 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:43.724152 (+ 93us) client_negotiation.cc:769] Sending connection context
0809 07:53:43.724436 (+ 284us) client_negotiation.cc:240] Negotiation successful
0809 07:53:43.724818 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":20,"thread_start_us":85,"threads_started":1}
Aug 09 07:53:43 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262022, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.62@KRBTEST.COM for oryx/127.28.112.61@KRBTEST.COM
I20260809 07:53:43.736531 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.728770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49126 (local address 127.28.112.61:42839)
0809 07:53:43.728865 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:43.728869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:43.729149 (+ 280us) server_negotiation.cc:407] Connection header received
0809 07:53:43.729317 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:43.729320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:43.729371 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:43.729436 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:43.730089 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.730577 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.731439 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.731650 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.731731 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.733912 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:43.733915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:43.733917 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:43.733937 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:43.735071 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.735549 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.735551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.735552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.735616 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.735843 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.735845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.735846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.735980 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:43.736046 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:43.736354 (+ 308us) server_negotiation.cc:299] Negotiation successful
0809 07:53:43.736433 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 07:53:43.736662 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.728745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:42839 (local address 127.0.0.1:49126)
0809 07:53:43.729035 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:43.729074 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:43.729197 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:43.729484 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:43.729490 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:43.729853 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:43.729981 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.729990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.730690 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.730694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.731331 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.731341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.731744 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.731748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.731850 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.732270 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:43.732281 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:43.733787 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:43.735168 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.735177 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.735183 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.735464 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.735682 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.735685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.735687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.735773 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.736058 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:43.736064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:43.736181 (+ 117us) client_negotiation.cc:769] Sending connection context
0809 07:53:43.736321 (+ 140us) client_negotiation.cc:240] Negotiation successful
0809 07:53:43.736497 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":73,"threads_started":1}
I20260809 07:53:43.737882 5902 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.112.61:42839: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bbbca9ad28064f5bab856505d7dff710"
member_type: NON_VOTER
last_known_addr {
host: "127.28.112.61"
port: 42839
}
attrs {
promote: true
}
}
dest_uuid: "64801c974cf64a69844c72bf20ac5983"
cas_config_opid_index: -1
I20260809 07:53:43.738595 5902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } }
I20260809 07:53:43.740134 6735 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER bbbca9ad28064f5bab856505d7dff710 (127.28.112.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } } }
I20260809 07:53:43.740944 6652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:43.741782 6735 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.112.61:42839
I20260809 07:53:43.742002 6735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } } }
I20260809 07:53:43.742087 6735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:43.742241 6652 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 1 FOLLOWER]: Refusing update from remote peer 64801c974cf64a69844c72bf20ac5983: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260809 07:53:43.743005 6735 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:43.743604 6592 master.cc:537] Master@127.28.112.61:42839 shutting down...
W20260809 07:53:43.745003 5867 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260809 07:53:43.746066 5867 consensus_peers.cc:487] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 -> Peer bbbca9ad28064f5bab856505d7dff710 (127.28.112.61:42839): Couldn't send request to peer bbbca9ad28064f5bab856505d7dff710. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 07:53:43.748157 6592 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:43.748409 6592 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:43.748502 6592 tablet_replica.cc:330] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: stopping tablet replica
WARNING: no policy specified for oryx/127.28.112.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.112.60@KRBTEST.COM" created.
I20260809 07:53:43.763650 6592 master.cc:552] Master@127.28.112.61:42839 shutdown complete.
I20260809 07:53:43.763705 6592 master_runner.cc:305] Clearing existing system tablet
I20260809 07:53:43.763950 6592 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:43.764012 6592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:43.765290 6592 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 07:53:43.766204 6744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:43.766427 6592 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:43.766530 6592 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:43.766561 6592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/wal
uuid: "bbbca9ad28064f5bab856505d7dff710"
format_stamp: "Formatted at 2026-08-09 07:53:43 on dist-test-slave-vpvm"
server_key: "d2f08c74a5eb08df8885dff80e49ad0f"
server_key_iv: "7f90b4c523094cbe01544cb2088e0479"
server_key_version: "encryptionkey@0"
I20260809 07:53:43.768316 6592 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:43.771328 6592 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 07:53:43.771401 6592 log.cc:1196] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260809 07:53:43.771648 6592 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: Deleting consensus metadata
I20260809 07:53:43.771796 6592 master_runner.cc:315] Copying system tablet from 127.28.112.62:37291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.112.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.60.keytab.
Entry for principal oryx/127.28.112.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.60.keytab.
I20260809 07:53:43.772737 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.60:41053
--webserver_interface=127.28.112.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.60.keytab
--principal=oryx/127.28.112.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:42033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:37291,127.28.112.61:42839,127.28.112.60:41053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 07:53:43.774806 6592 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: tablet copy: Beginning tablet copy session from remote peer at address 127.28.112.62:37291
I20260809 07:53:43.783569 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.775717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49500 (local address 127.28.112.62:37291)
0809 07:53:43.775825 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:43.775829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:43.775842 (+ 13us) server_negotiation.cc:407] Connection header received
0809 07:53:43.775883 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:43.775887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:43.775931 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:43.776013 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:43.776544 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.777362 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.778510 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.778848 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.778968 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.780199 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:43.780204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:43.780207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:43.780233 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:43.781829 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.782338 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.782342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.782344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.782430 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.782706 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.782709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.782711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.782910 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:43.783009 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:43.783307 (+ 298us) server_negotiation.cc:299] Negotiation successful
0809 07:53:43.783417 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 07:53:43.783584 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.775382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:49500)
0809 07:53:43.775647 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:43.775662 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:43.775770 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:43.776049 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:43.776052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:43.776259 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:43.776397 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.776406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.777485 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.777489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.778392 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.778405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.778984 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.778988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.779081 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.779620 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:43.779635 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:43.780053 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:43.781991 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.781995 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.781997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.782243 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.782523 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.782526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.782528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.782620 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.783013 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:43.783016 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:43.783119 (+ 103us) client_negotiation.cc:769] Sending connection context
0809 07:53:43.783254 (+ 135us) client_negotiation.cc:240] Negotiation successful
0809 07:53:43.783389 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
I20260809 07:53:43.784459 5930 tablet_copy_service.cc:140] P 64801c974cf64a69844c72bf20ac5983: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bbbca9ad28064f5bab856505d7dff710 ({username='oryx', principal='oryx/127.28.112.61@KRBTEST.COM'} at 127.0.0.1:49500)
I20260809 07:53:43.784634 5930 tablet_copy_service.cc:161] P 64801c974cf64a69844c72bf20ac5983: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bbbca9ad28064f5bab856505d7dff710 at {username='oryx', principal='oryx/127.28.112.61@KRBTEST.COM'} at 127.0.0.1:49500: session id = bbbca9ad28064f5bab856505d7dff710-00000000000000000000000000000000
I20260809 07:53:43.786490 5930 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 07:53:43.788378 6592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 07:53:43.790673 6592 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: tablet copy: Starting download of 0 data blocks...
I20260809 07:53:43.790853 6592 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: tablet copy: Starting download of 1 WAL segments...
I20260809 07:53:43.792557 6592 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 07:53:43.794229 5930 tablet_copy_service.cc:342] P 64801c974cf64a69844c72bf20ac5983: Request end of tablet copy session bbbca9ad28064f5bab856505d7dff710-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.112.61@KRBTEST.COM'} at 127.0.0.1:49500
I20260809 07:53:43.794338 5930 tablet_copy_service.cc:433] P 64801c974cf64a69844c72bf20ac5983: ending tablet copy session bbbca9ad28064f5bab856505d7dff710-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bbbca9ad28064f5bab856505d7dff710
I20260809 07:53:43.802035 6592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:43.803646 6756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:43.806771 6757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:43.807149 6759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:43.810755 6592 server_base.cc:1034] running on GCE node
I20260809 07:53:43.810961 6592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:43.811302 6592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:43.812491 6592 hybrid_clock.cc:648] HybridClock initialized: now 1786262023812478 us; error 30 us; skew 500 ppm
Aug 09 07:53:43 dist-test-slave-vpvm krb5kdc[5823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262023, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:43.814108 6592 init.cc:377] Logged in from keytab as oryx/127.28.112.61@KRBTEST.COM (short username oryx)
I20260809 07:53:43.814602 6592 webserver.cc:460] Webserver started at http://127.28.112.61:45393/ using document root <none> and password file <none>
I20260809 07:53:43.814782 6592 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:43.814822 6592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:43.816116 6592 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 07:53:43.817493 6772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:43.817768 6592 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 07:53:43.817845 6592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/wal
uuid: "bbbca9ad28064f5bab856505d7dff710"
format_stamp: "Formatted at 2026-08-09 07:53:43 on dist-test-slave-vpvm"
server_key: "d2f08c74a5eb08df8885dff80e49ad0f"
server_key_iv: "7f90b4c523094cbe01544cb2088e0479"
server_key_version: "encryptionkey@0"
I20260809 07:53:43.817929 6592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:43.842614 6592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:43.844695 6592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:43.849575 6592 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.61:42839
I20260809 07:53:43.849673 6849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.61:42839 every 8 connection(s)
I20260809 07:53:43.850190 6592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-1/data/info.pb
I20260809 07:53:43.852006 6851 sys_catalog.cc:263] Verifying existing consensus state
I20260809 07:53:43.853075 6851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: Bootstrap starting.
I20260809 07:53:43.863428 6851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 07:53:43.863709 6851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: Bootstrap complete.
I20260809 07:53:43.864251 6851 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } }
I20260809 07:53:43.864382 6851 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bbbca9ad28064f5bab856505d7dff710, State: Initialized, Role: LEARNER
I20260809 07:53:43.864482 6851 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } }
I20260809 07:53:43.864797 6854 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } } }
I20260809 07:53:43.864890 6851 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:43.865017 6854 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [sys.catalog]: This master's current role is: LEARNER
I20260809 07:53:43.867748 6872 catalog_manager.cc:1259] Loaded cluster ID: 4b926155880248228e380cab0d664580
I20260809 07:53:43.867862 6872 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: loading cluster ID for follower catalog manager: success
I20260809 07:53:43.870051 6872 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: acquiring CA information for follower catalog manager: success
I20260809 07:53:43.871430 6872 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 07:53:43.900334 6746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:43.900593 6746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:43.900660 6746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:43.904651 6746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:43.904702 6746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:43.904726 6746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:43.904742 6746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:43.904760 6746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:53:43.909775 6746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:42033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:37291,127.28.112.61:42839,127.28.112.60:41053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.60.keytab
--principal=oryx/127.28.112.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.60:41053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.112.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:43.910754 6746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:43.911676 6746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 07:53:43.919739 6746 server_base.cc:1034] running on GCE node
W20260809 07:53:43.919689 6880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:43.919689 6883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:43.919704 6879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:43.920454 6746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:43.921514 6746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:43.922667 6746 hybrid_clock.cc:648] HybridClock initialized: now 1786262023922650 us; error 30 us; skew 500 ppm
Aug 09 07:53:43 dist-test-slave-vpvm krb5kdc[5823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262023, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:43.925347 6746 init.cc:377] Logged in from keytab as oryx/127.28.112.60@KRBTEST.COM (short username oryx)
I20260809 07:53:43.926260 6746 webserver.cc:460] Webserver started at http://127.28.112.60:41325/ using document root <none> and password file <none>
I20260809 07:53:43.926790 6746 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:43.926851 6746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:43.927043 6746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:43.929128 6746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/data/instance:
uuid: "add5262818464f139ffb4c5bade4a733"
format_stamp: "Formatted at 2026-08-09 07:53:43 on dist-test-slave-vpvm"
server_key: "290fcc5c862edca49d5864c2b9269bee"
server_key_iv: "7fcd63e610b5563c58677f7a15762c80"
server_key_version: "encryptionkey@0"
I20260809 07:53:43.929698 6746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/wal/instance:
uuid: "add5262818464f139ffb4c5bade4a733"
format_stamp: "Formatted at 2026-08-09 07:53:43 on dist-test-slave-vpvm"
server_key: "290fcc5c862edca49d5864c2b9269bee"
server_key_iv: "7fcd63e610b5563c58677f7a15762c80"
server_key_version: "encryptionkey@0"
I20260809 07:53:43.933367 6746 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 07:53:43.935767 6893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:43.937016 6746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 07:53:43.937124 6746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/wal
uuid: "add5262818464f139ffb4c5bade4a733"
format_stamp: "Formatted at 2026-08-09 07:53:43 on dist-test-slave-vpvm"
server_key: "290fcc5c862edca49d5864c2b9269bee"
server_key_iv: "7fcd63e610b5563c58677f7a15762c80"
server_key_version: "encryptionkey@0"
I20260809 07:53:43.937238 6746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:43.951762 6746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:43.954710 6746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:43.954877 6746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:43.961306 6746 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.60:41053
I20260809 07:53:43.961603 6972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.60:41053 every 8 connection(s)
I20260809 07:53:43.962280 6746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/data/info.pb
I20260809 07:53:43.965027 6973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:43.968811 6973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } has no permanent_uuid. Determining permanent_uuid...
I20260809 07:53:43.969159 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 6746
Aug 09 07:53:43 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262023, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.60@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM
Aug 09 07:53:43 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.112.60@KRBTEST.COM
I20260809 07:53:43.981510 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.971497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49516 (local address 127.28.112.62:37291)
0809 07:53:43.971634 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:43.971639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:43.971913 (+ 274us) server_negotiation.cc:407] Connection header received
0809 07:53:43.972203 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:43.972206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:43.972256 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:43.972331 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:43.973290 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.973809 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.975091 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.975331 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.975424 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.977795 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:43.977799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:43.977801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:43.977827 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:43.979363 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.979954 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.979958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.979960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.980049 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.980430 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.980434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.980436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.980674 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:43.980755 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:43.981242 (+ 487us) server_negotiation.cc:299] Negotiation successful
0809 07:53:43.981355 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 07:53:43.982319 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.971276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:49516)
0809 07:53:43.971779 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:43.971825 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:43.972040 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:43.972395 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:43.972402 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:43.972844 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:43.973158 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.973170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.973929 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.973932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.974960 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.974971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.975475 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.975480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.975631 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.976122 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:43.976144 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:43.977618 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:43.979468 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.979476 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.979484 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.979851 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.980130 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.980133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.980135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.980303 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.980777 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:43.980784 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:43.981018 (+ 234us) client_negotiation.cc:769] Sending connection context
0809 07:53:43.981175 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 07:53:43.981473 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":88,"threads_started":1}
I20260809 07:53:43.982324 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.970595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.28.112.60:41053)
0809 07:53:43.970955 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:43.970971 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:43.971002 (+ 31us) server_negotiation.cc:407] Connection header received
0809 07:53:43.971716 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:43.971736 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:43.972043 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:43.972240 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:43.973125 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.974244 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.975265 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.975610 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.975702 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.978206 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:43.978213 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:43.978225 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:43.978250 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:43.979975 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.980470 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.980476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.980478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.980556 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.980834 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.980837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.980838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.981072 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:43.981242 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:43.981482 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 07:53:43.981616 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":108,"threads_started":1}
I20260809 07:53:43.983493 6973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 07:53:43 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262023, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.60@KRBTEST.COM for oryx/127.28.112.61@KRBTEST.COM
I20260809 07:53:43.990906 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.983935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:42839 (local address 127.0.0.1:49138)
0809 07:53:43.984248 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:43.984259 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:43.984314 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:43.984507 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:43.984509 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:43.984627 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:43.984718 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.984724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.985547 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.985550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.986124 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.986133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.986586 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.986588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.986657 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.987035 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:43.987046 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:43.988158 (+ 1112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:43.989747 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.989750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.989751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.989935 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.990182 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.990184 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.990185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.990253 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.990563 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:43.990565 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:43.990638 (+ 73us) client_negotiation.cc:769] Sending connection context
0809 07:53:43.990717 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 07:53:43.990806 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250}
I20260809 07:53:43.990978 6977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.984073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49138 (local address 127.28.112.61:42839)
0809 07:53:43.984291 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:43.984294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:43.984306 (+ 12us) server_negotiation.cc:407] Connection header received
0809 07:53:43.984381 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:43.984383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:43.984419 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:43.984490 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:43.984803 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.985450 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.986244 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.986485 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.986570 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.988342 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:43.988345 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:43.988347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:43.988371 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:43.989645 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.990036 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.990038 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.990039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.990116 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.990311 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.990313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.990314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.990474 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:43.990548 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:43.990758 (+ 210us) server_negotiation.cc:299] Negotiation successful
0809 07:53:43.990849 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":81,"threads_started":1}
I20260809 07:53:43.991581 6973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 07:53:43 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262023, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.60@KRBTEST.COM for oryx/127.28.112.60@KRBTEST.COM
I20260809 07:53:43.998072 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.991974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:41053 (local address 127.0.0.1:42062)
0809 07:53:43.992089 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:43.992100 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:43.992155 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:43.992316 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:43.992318 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:43.992450 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:43.992512 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.992516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.993134 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.993137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.993696 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:43.993706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.994176 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.994180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:43.994260 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.994622 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:43.994643 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:43.995687 (+ 1044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:43.996978 (+ 1291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.996981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.996982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.997160 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.997355 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:43.997357 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:43.997358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:43.997420 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:43.997711 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:43.997713 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:43.997783 (+ 70us) client_negotiation.cc:769] Sending connection context
0809 07:53:43.997861 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 07:53:43.997945 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 07:53:43.998169 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:43.992044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42062 (local address 127.28.112.60:41053)
0809 07:53:43.992129 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:43.992132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:43.992143 (+ 11us) server_negotiation.cc:407] Connection header received
0809 07:53:43.992209 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:43.992210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:43.992241 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:43.992298 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:43.992587 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.993042 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.993798 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:43.994064 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:43.994167 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:43.995805 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:43.995808 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:43.995810 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:43.995830 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:43.996878 (+ 1048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.997231 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.997233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.997234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.997296 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:43.997478 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:43.997479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:43.997480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:43.997635 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:43.997713 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:43.997902 (+ 189us) server_negotiation.cc:299] Negotiation successful
0809 07:53:43.998030 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":32}
I20260809 07:53:44.001350 6973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: Bootstrap starting.
I20260809 07:53:44.003659 6973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:44.004356 6973 log.cc:824] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:44.006207 6973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: No bootstrap required, opened a new log
I20260809 07:53:44.009181 6973 raft_consensus.cc:348] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } }
I20260809 07:53:44.009378 6973 raft_consensus.cc:374] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:44.009428 6973 raft_consensus.cc:729] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add5262818464f139ffb4c5bade4a733, State: Initialized, Role: FOLLOWER
I20260809 07:53:44.009814 6973 consensus_queue.cc:260] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } }
I20260809 07:53:44.010285 6979 sys_catalog.cc:455] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } } }
I20260809 07:53:44.010397 6979 sys_catalog.cc:458] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:53:44.010969 6973 sys_catalog.cc:564] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 07:53:44.015324 7000 catalog_manager.cc:1558] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:53:44.015390 7000 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:53:44.018723 6746 master_runner.cc:428] Detected that this master add5262818464f139ffb4c5bade4a733 is joining an existing cluster
I20260809 07:53:44.018785 6746 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.112.60:41053
I20260809 07:53:44.035267 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 5990
I20260809 07:53:44.053834 6977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.021073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.28.112.61:42839)
0809 07:53:44.021193 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.021197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.021360 (+ 163us) server_negotiation.cc:407] Connection header received
0809 07:53:44.021462 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.021465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.021512 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.021583 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:44.022081 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.022835 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.023696 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.024016 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.024123 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.025202 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:44.025205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:44.025208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:44.025233 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:44.026706 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.050792 (+ 24086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.050797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.050799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.050905 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.051241 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.051245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.051246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.051494 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:44.051572 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.053501 (+ 1929us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.053670 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 07:53:44.053171 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.021729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.28.112.60:41053)
0809 07:53:44.021821 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.021825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.022231 (+ 406us) server_negotiation.cc:407] Connection header received
0809 07:53:44.022357 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.022360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.022390 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.022428 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:44.022945 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.023697 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.024579 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.024896 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.025005 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.026152 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:44.026155 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:44.026158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:44.026180 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:44.027952 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.051784 (+ 23832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.051787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.051789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.051877 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.052254 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.052258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.052259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.052471 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:44.052554 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.052867 (+ 313us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.052990 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":35}
I20260809 07:53:44.054405 7008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.021023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:42839 (local address 127.0.0.1:49142)
0809 07:53:44.021286 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.021298 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.021389 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.021608 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.021611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.021804 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:44.021927 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.021933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.022957 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.022960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.023573 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.023581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.024147 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.024151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.024244 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.024753 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:44.024765 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:44.025068 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:44.026803 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.026807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.026809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.050623 (+ 23814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.051028 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.051031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.051033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.051126 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.051663 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:44.051666 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:44.053306 (+ 1640us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.054087 (+ 781us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.054242 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":21,"thread_start_us":100,"threads_started":1}
I20260809 07:53:44.054831 7010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.021430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:41053 (local address 127.0.0.1:42076)
0809 07:53:44.022157 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.022171 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.022280 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.022525 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.022528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.022727 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:44.022820 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.022826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.023793 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.023796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.024466 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.024477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.025104 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.025107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.025192 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.025589 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:44.025603 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:44.026029 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:44.028058 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.028061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.028063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.050726 (+ 22663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.052027 (+ 1301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.052030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.052032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.052150 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.052599 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:44.052601 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:44.052708 (+ 107us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.054534 (+ 1826us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.054671 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":97,"threads_started":1}
I20260809 07:53:44.056039 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.022258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49524 (local address 127.28.112.62:37291)
0809 07:53:44.022363 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.022368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.026917 (+ 4549us) server_negotiation.cc:407] Connection header received
0809 07:53:44.027079 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.027083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.027136 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.027211 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:44.027646 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.028125 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.028875 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.029087 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.029171 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.051386 (+ 22215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:44.051390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:44.051393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:44.051421 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:44.053116 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.054700 (+ 1584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.054704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.054706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.054792 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.055103 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.055107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.055108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.055332 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:44.055417 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.055789 (+ 372us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.055903 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 07:53:44.056465 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.022372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:49524)
0809 07:53:44.026830 (+ 4458us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.026844 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.026983 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.027241 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.027243 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.027419 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:44.027518 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.027523 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.028226 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.028229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.028769 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.028777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.029184 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.029188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.029271 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.029647 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:44.029662 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:44.051224 (+ 21562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:44.054349 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.054352 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.054354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.054575 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.054910 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.054912 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.054914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.055000 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.055480 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:44.055483 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:44.055579 (+ 96us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.056187 (+ 608us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.056321 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3695,"thread_start_us":89,"threads_started":1}
I20260809 07:53:44.059018 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:43077
--local_ip_for_outbound_sockets=127.28.112.1
--tserver_master_addrs=127.28.112.62:37291
--webserver_port=45283
--webserver_interface=127.28.112.1
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.1.keytab
--principal=oryx/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:42033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.112.62:37291,127.28.112.61:42839
--tserver_master_addrs=127.28.112.62:37291,127.28.112.61:42839,127.28.112.60:41053 with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 07:53:44 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262022, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.62@KRBTEST.COM for oryx/127.28.112.60@KRBTEST.COM
I20260809 07:53:44.067646 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.060439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:41053 (local address 127.0.0.1:42078)
0809 07:53:44.060572 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.060588 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.060684 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.060924 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.060928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.061087 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:44.061170 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.061176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.061846 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.061850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.062424 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.062433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.062835 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.062839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.062926 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.063313 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:44.063328 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:44.064563 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:44.066180 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.066183 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.066186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.066373 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.066632 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.066635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.066637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.066792 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.067132 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:44.067143 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:44.067230 (+ 87us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.067314 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.067490 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 07:53:44.067654 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.060488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.28.112.60:41053)
0809 07:53:44.060585 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.060589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.060639 (+ 50us) server_negotiation.cc:407] Connection header received
0809 07:53:44.060777 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.060779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.060820 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.060891 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:44.061274 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.061748 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.062532 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.062738 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.062824 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.064695 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:44.064698 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:44.064701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:44.064724 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:44.066036 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.066463 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.066467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.066469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.066549 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.066879 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.066882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.066884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.067044 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:44.067128 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.067409 (+ 281us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.067507 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 07:53:44.068750 5902 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.112.60:41053: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "add5262818464f139ffb4c5bade4a733"
member_type: NON_VOTER
last_known_addr {
host: "127.28.112.60"
port: 41053
}
attrs {
promote: true
}
}
dest_uuid: "64801c974cf64a69844c72bf20ac5983"
cas_config_opid_index: 5
I20260809 07:53:44.069355 5902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } }
I20260809 07:53:44.070393 6734 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER add5262818464f139ffb4c5bade4a733 (127.28.112.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } } }
I20260809 07:53:44.071557 6734 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.112.60:41053
I20260809 07:53:44.071736 6734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } } }
I20260809 07:53:44.071838 6734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:44.072759 6935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:44.073972 6935 raft_consensus.cc:1261] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 1 FOLLOWER]: Refusing update from remote peer 64801c974cf64a69844c72bf20ac5983: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 07:53:44.074566 6735 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [LEADER]: Connected to new peer: Peer: permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:44.078627 6935 raft_consensus.cc:858] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } }
W20260809 07:53:44.078842 6935 raft_consensus.cc:1565] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } }
I20260809 07:53:44.079870 6977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.072424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49146 (local address 127.28.112.61:42839)
0809 07:53:44.072538 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.072543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.075165 (+ 2622us) server_negotiation.cc:407] Connection header received
0809 07:53:44.075364 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.075368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.075434 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.075504 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:44.075880 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.076595 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.078459 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.079077 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.079175 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.079325 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.079682 (+ 357us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.079755 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 07:53:44.080260 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.072210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:42839 (local address 127.0.0.1:49146)
0809 07:53:44.074470 (+ 2260us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.074485 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.075238 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.075655 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.075658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.075677 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:44.075751 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.075757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.076930 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.076933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.078308 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.078320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.079425 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.079429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.079518 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.079563 (+ 45us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.080064 (+ 501us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.080150 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2195}
I20260809 07:53:44.080756 6812 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 1 LEARNER]: Refusing update from remote peer 64801c974cf64a69844c72bf20ac5983: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 07:53:44.080756 6935 raft_consensus.cc:2932] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER bbbca9ad28064f5bab856505d7dff710 (127.28.112.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } } }
I20260809 07:53:44.081267 6735 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 07:53:44.082676 7020 sys_catalog.cc:455] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } } }
I20260809 07:53:44.082849 7020 sys_catalog.cc:458] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260809 07:53:44.082988 7020 sys_catalog.cc:455] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } } }
I20260809 07:53:44.083114 7020 sys_catalog.cc:458] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260809 07:53:44.083099 6812 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER add5262818464f139ffb4c5bade4a733 (127.28.112.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } } }
I20260809 07:53:44.084890 6854 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64801c974cf64a69844c72bf20ac5983. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } } }
I20260809 07:53:44.085005 6854 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [sys.catalog]: This master's current role is: LEARNER
I20260809 07:53:44.085103 6854 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } } }
I20260809 07:53:44.085197 6854 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [sys.catalog]: This master's current role is: LEARNER
I20260809 07:53:44.085662 6979 sys_catalog.cc:455] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64801c974cf64a69844c72bf20ac5983. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } } }
I20260809 07:53:44.085749 6979 sys_catalog.cc:458] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260809 07:53:44.086205 6735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: NON_VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: true } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } } }
I20260809 07:53:44.086316 6735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:44.086400 6746 master.cc:537] Master@127.28.112.60:41053 shutting down...
W20260809 07:53:44.087505 5867 consensus_peers.cc:487] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 -> Peer add5262818464f139ffb4c5bade4a733 (127.28.112.60:41053): Couldn't send request to peer add5262818464f139ffb4c5bade4a733. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 07:53:44.088987 6734 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983: attempting to promote NON_VOTER bbbca9ad28064f5bab856505d7dff710 to VOTER
I20260809 07:53:44.089391 6734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } }
I20260809 07:53:44.090960 6746 raft_consensus.cc:2229] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260809 07:53:44.091112 6810 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 1 LEARNER]: Refusing update from remote peer 64801c974cf64a69844c72bf20ac5983: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260809 07:53:44.091192 6746 raft_consensus.cc:2258] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260809 07:53:44.091281 6746 tablet_replica.cc:330] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: stopping tablet replica
I20260809 07:53:44.091629 6735 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260809 07:53:44.092160 5867 consensus_peers.cc:487] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 -> Peer add5262818464f139ffb4c5bade4a733 (127.28.112.60:41053): Couldn't send request to peer add5262818464f139ffb4c5bade4a733. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 07:53:44.093338 6734 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bbbca9ad28064f5bab856505d7dff710 (127.28.112.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } } }
I20260809 07:53:44.094856 6734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } } }
I20260809 07:53:44.094959 6734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:44.095088 6735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } } }
I20260809 07:53:44.095160 6735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:44.095379 6812 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bbbca9ad28064f5bab856505d7dff710 (127.28.112.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } } }
I20260809 07:53:44.096566 7024 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } } }
I20260809 07:53:44.096698 7024 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:53:44.114609 6746 master.cc:552] Master@127.28.112.60:41053 shutdown complete.
I20260809 07:53:44.114675 6746 master_runner.cc:305] Clearing existing system tablet
I20260809 07:53:44.114909 6746 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:44.114979 6746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:44.116178 6746 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260809 07:53:44.117056 7032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:44.117234 6746 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:44.117341 6746 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:44.117372 6746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/wal
uuid: "add5262818464f139ffb4c5bade4a733"
format_stamp: "Formatted at 2026-08-09 07:53:43 on dist-test-slave-vpvm"
server_key: "290fcc5c862edca49d5864c2b9269bee"
server_key_iv: "7fcd63e610b5563c58677f7a15762c80"
server_key_version: "encryptionkey@0"
I20260809 07:53:44.119163 6746 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:44.121448 6746 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 07:53:44.121533 6746 log.cc:1196] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260809 07:53:44.121805 6746 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: Deleting consensus metadata
I20260809 07:53:44.122026 6746 master_runner.cc:315] Copying system tablet from 127.28.112.62:37291
I20260809 07:53:44.123111 6746 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: tablet copy: Beginning tablet copy session from remote peer at address 127.28.112.62:37291
I20260809 07:53:44.132126 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.123664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:49530)
0809 07:53:44.124045 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.124060 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.124186 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.124515 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.124518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.124726 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:44.124828 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.124835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.125880 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.125884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.126778 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.126789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.127371 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.127374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.127475 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.127990 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:44.128004 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:44.128397 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:44.130500 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.130504 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.130506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.130748 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.131079 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.131082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.131084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.131185 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.131592 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:44.131595 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:44.131696 (+ 101us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.131805 (+ 109us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.131955 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":87,"threads_started":1}
I20260809 07:53:44.132145 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.123765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49530 (local address 127.28.112.62:37291)
0809 07:53:44.123895 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.123900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.124165 (+ 265us) server_negotiation.cc:407] Connection header received
0809 07:53:44.124331 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.124334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.124393 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.124483 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:44.124946 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.125748 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.126915 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.127248 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.127362 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.128555 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:44.128559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:44.128562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:44.128592 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:44.130367 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.130899 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.130902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.130904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.130994 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.131273 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.131276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.131278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.131487 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:44.131583 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.131865 (+ 282us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.131984 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 07:53:44.132885 5930 tablet_copy_service.cc:140] P 64801c974cf64a69844c72bf20ac5983: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer add5262818464f139ffb4c5bade4a733 ({username='oryx', principal='oryx/127.28.112.60@KRBTEST.COM'} at 127.0.0.1:49530)
I20260809 07:53:44.132980 5930 tablet_copy_service.cc:161] P 64801c974cf64a69844c72bf20ac5983: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer add5262818464f139ffb4c5bade4a733 at {username='oryx', principal='oryx/127.28.112.60@KRBTEST.COM'} at 127.0.0.1:49530: session id = add5262818464f139ffb4c5bade4a733-00000000000000000000000000000000
I20260809 07:53:44.134049 5930 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 07:53:44.135326 6746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 07:53:44.137514 6746 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: tablet copy: Starting download of 0 data blocks...
I20260809 07:53:44.137688 6746 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: tablet copy: Starting download of 1 WAL segments...
I20260809 07:53:44.139299 6746 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 07:53:44.141149 5930 tablet_copy_service.cc:342] P 64801c974cf64a69844c72bf20ac5983: Request end of tablet copy session add5262818464f139ffb4c5bade4a733-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.112.60@KRBTEST.COM'} at 127.0.0.1:49530
I20260809 07:53:44.141232 5930 tablet_copy_service.cc:433] P 64801c974cf64a69844c72bf20ac5983: ending tablet copy session add5262818464f139ffb4c5bade4a733-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer add5262818464f139ffb4c5bade4a733
I20260809 07:53:44.148537 6746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:44.150231 7042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:44.150236 7044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:44.150244 7049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:44.150830 6746 server_base.cc:1034] running on GCE node
I20260809 07:53:44.151039 6746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:44.151343 6746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:44.152500 6746 hybrid_clock.cc:648] HybridClock initialized: now 1786262024152485 us; error 33 us; skew 500 ppm
Aug 09 07:53:44 dist-test-slave-vpvm krb5kdc[5823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262024, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:44.154350 6746 init.cc:377] Logged in from keytab as oryx/127.28.112.60@KRBTEST.COM (short username oryx)
I20260809 07:53:44.154831 6746 webserver.cc:460] Webserver started at http://127.28.112.60:38213/ using document root <none> and password file <none>
I20260809 07:53:44.155006 6746 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:44.155045 6746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:44.156265 6746 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:44.157290 7058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:44.157578 6746 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:44.157665 6746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/wal
uuid: "add5262818464f139ffb4c5bade4a733"
format_stamp: "Formatted at 2026-08-09 07:53:43 on dist-test-slave-vpvm"
server_key: "290fcc5c862edca49d5864c2b9269bee"
server_key_iv: "7fcd63e610b5563c58677f7a15762c80"
server_key_version: "encryptionkey@0"
I20260809 07:53:44.157750 6746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:44.178264 6746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:44.180266 6746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:44.185281 6746 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.60:41053
I20260809 07:53:44.185675 7130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.60:41053 every 8 connection(s)
I20260809 07:53:44.186007 6746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/master-2/data/info.pb
I20260809 07:53:44.187696 7131 sys_catalog.cc:263] Verifying existing consensus state
I20260809 07:53:44.188660 7131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: Bootstrap starting.
I20260809 07:53:44.195178 7131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 07:53:44.195405 7131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: Bootstrap complete.
I20260809 07:53:44.195842 7131 raft_consensus.cc:348] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } }
I20260809 07:53:44.195955 7131 raft_consensus.cc:729] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: add5262818464f139ffb4c5bade4a733, State: Initialized, Role: LEARNER
I20260809 07:53:44.196063 7131 consensus_queue.cc:260] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } }
I20260809 07:53:44.196346 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } } }
I20260809 07:53:44.196426 7131 sys_catalog.cc:564] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:44.196480 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: This master's current role is: LEARNER
I20260809 07:53:44.199395 7154 catalog_manager.cc:1259] Loaded cluster ID: 4b926155880248228e380cab0d664580
I20260809 07:53:44.199451 7154 catalog_manager.cc:1552] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: loading cluster ID for follower catalog manager: success
I20260809 07:53:44.200937 7154 catalog_manager.cc:1574] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: acquiring CA information for follower catalog manager: success
I20260809 07:53:44.202040 7154 catalog_manager.cc:1602] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 07:53:44.203305 7012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:44.203517 7012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:44.203565 7012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:44.207517 7012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:44.207567 7012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:44.207639 7012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:53:44.212728 7012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:42033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.1.keytab
--principal=oryx/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:43077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=45283
--tserver_master_addrs=127.28.112.62:37291,127.28.112.61:42839,127.28.112.60:41053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:44.213702 7012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:44.214735 7012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:44.224097 7159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:44.224071 7166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:44.224366 7012 server_base.cc:1034] running on GCE node
W20260809 07:53:44.224067 7162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:44.224866 7012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:44.225454 7012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:44.226622 7012 hybrid_clock.cc:648] HybridClock initialized: now 1786262024226589 us; error 47 us; skew 500 ppm
Aug 09 07:53:44 dist-test-slave-vpvm krb5kdc[5823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262024, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:44.229161 7012 init.cc:377] Logged in from keytab as oryx/127.28.112.1@KRBTEST.COM (short username oryx)
I20260809 07:53:44.230039 7012 webserver.cc:460] Webserver started at http://127.28.112.1:45283/ using document root <none> and password file <none>
I20260809 07:53:44.230589 7012 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:44.230685 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:44.234289 7012 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260809 07:53:44.236196 7174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:44.237336 7012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:44.237440 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "730d23ec728547609406d0fceaf73197"
format_stamp: "Formatted at 2026-08-09 07:53:42 on dist-test-slave-vpvm"
server_key: "c4e7ea4086348b68398c3bdeaad1f108"
server_key_iv: "3b1ffe4e188887d4ccdaa73f67531b5e"
server_key_version: "encryptionkey@0"
I20260809 07:53:44.237926 7012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:44.244597 6388 heartbeater.cc:503] Master 127.28.112.62:37291 was elected leader, sending a full tablet report...
I20260809 07:53:44.262548 7012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:44.265476 7012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:44.265697 7012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:44.266368 7012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:44.267264 7012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:44.267315 7012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:44.267371 7012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:44.267400 7012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:44.276068 7012 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:43077
I20260809 07:53:44.276096 7348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:43077 every 8 connection(s)
I20260809 07:53:44.277040 7012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:53:44.286604 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 7012
I20260809 07:53:44.286808 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 6191
Aug 09 07:53:44 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262024, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.60@KRBTEST.COM
Aug 09 07:53:44 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262024, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM
I20260809 07:53:44.292742 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.278892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:38317 (local address 127.28.112.62:37291)
0809 07:53:44.279036 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.279040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.280376 (+ 1336us) server_negotiation.cc:407] Connection header received
0809 07:53:44.281417 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.281421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.281481 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.281536 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:44.282849 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.283377 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.284907 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.285207 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.285303 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.289056 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:44.289059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:44.289061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:44.289085 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:44.290697 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.291320 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.291324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.291325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.291399 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.291713 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.291716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.291717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.291892 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:44.291959 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.292500 (+ 541us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.292603 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 07:53:44.293115 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.278892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:52661 (local address 127.28.112.60:41053)
0809 07:53:44.279151 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.279154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.280433 (+ 1279us) server_negotiation.cc:407] Connection header received
0809 07:53:44.281446 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.281449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.281501 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.281558 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:44.282858 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.283852 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.284729 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.285040 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.285151 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.288794 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:44.288799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:44.288802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:44.288834 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:44.290613 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.291425 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.291430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.291432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.291548 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.291935 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.291939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.291942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.292231 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:44.292321 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.292767 (+ 446us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.292923 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":83,"threads_started":1}
I20260809 07:53:44.294134 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.279403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.1:38317)
0809 07:53:44.280202 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.280261 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.281185 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.281689 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.281692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.282129 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:44.282694 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.282703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.283522 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.283525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.284748 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.284759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.285306 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.285309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.285573 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.286398 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:44.286419 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:44.288935 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:44.290811 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.290814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.290815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.291227 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.291487 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.291490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.291491 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.291632 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.292026 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:44.292032 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:44.292275 (+ 243us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.292941 (+ 666us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.293284 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":170,"spinlock_wait_cycles":60672,"thread_start_us":157,"threads_started":1}
I20260809 07:53:44.294469 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.279403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:41053 (local address 127.28.112.1:52661)
0809 07:53:44.280203 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.280261 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.281188 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.281689 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.281692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.282143 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:44.282696 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.282703 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.283953 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.283956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.284602 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.284611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.285246 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.285252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.285578 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.286479 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:44.286491 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:44.288612 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:44.290768 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.290776 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.290794 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.291279 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.291683 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.291686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.291689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.291802 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.292414 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:44.292418 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:44.292527 (+ 109us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.293817 (+ 1290us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.294015 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":112,"spinlock_wait_cycles":11008,"thread_start_us":152,"threads_started":1}
I20260809 07:53:44.296433 7350 heartbeater.cc:347] Connected to a master server at 127.28.112.62:37291
I20260809 07:53:44.296689 7350 heartbeater.cc:464] Registering TS with master...
Aug 09 07:53:44 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262024, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.61@KRBTEST.COM
I20260809 07:53:44.297443 7350 heartbeater.cc:511] Master 127.28.112.62:37291 requested a full tablet report, sending...
I20260809 07:53:44.298840 5902 ts_manager.cc:194] Re-registered known tserver with Master: 730d23ec728547609406d0fceaf73197 (127.28.112.1:43077)
I20260809 07:53:44.299602 5902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:38317
I20260809 07:53:44.301306 6977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.280861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:51207 (local address 127.28.112.61:42839)
0809 07:53:44.280973 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.280977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.280988 (+ 11us) server_negotiation.cc:407] Connection header received
0809 07:53:44.281374 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.281378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.281437 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.281508 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:44.282819 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.283642 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.285418 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.285741 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.285844 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.297825 (+ 11981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:44.297830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:44.297833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:44.297863 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:44.299575 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.300108 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.300111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.300113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.300207 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.300500 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.300504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.300505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.300719 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:44.300811 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.301058 (+ 247us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.301168 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 07:53:44.301326 7358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.280503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:42839 (local address 127.28.112.1:51207)
0809 07:53:44.280756 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.280768 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.281185 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.281682 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.281689 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.282142 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:44.282680 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.282694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.283757 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.283760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.284764 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.284771 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.285897 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.285901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.286013 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.286399 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:44.286438 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:44.297658 (+ 11220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:44.299751 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.299754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.299756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.299992 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.300311 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.300313 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.300314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.300382 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.300813 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:44.300816 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:44.300909 (+ 93us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.300997 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.301147 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":85,"spinlock_wait_cycles":768,"thread_start_us":68,"threads_started":1}
I20260809 07:53:44.302822 7349 heartbeater.cc:347] Connected to a master server at 127.28.112.60:41053
I20260809 07:53:44.302929 7349 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:44.303030 7351 heartbeater.cc:347] Connected to a master server at 127.28.112.61:42839
I20260809 07:53:44.303099 7351 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:44.303128 7349 heartbeater.cc:511] Master 127.28.112.60:41053 requested a full tablet report, sending...
I20260809 07:53:44.303212 7351 heartbeater.cc:511] Master 127.28.112.61:42839 requested a full tablet report, sending...
I20260809 07:53:44.303555 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:39549
--local_ip_for_outbound_sockets=127.28.112.2
--tserver_master_addrs=127.28.112.62:37291
--webserver_port=34257
--webserver_interface=127.28.112.2
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.2.keytab
--principal=oryx/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:42033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.112.62:37291,127.28.112.61:42839
--tserver_master_addrs=127.28.112.62:37291,127.28.112.61:42839,127.28.112.60:41053 with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 07:53:44.304195 7081 ts_manager.cc:194] Registered new tserver with Master: 730d23ec728547609406d0fceaf73197 (127.28.112.1:43077)
I20260809 07:53:44.304311 6799 ts_manager.cc:194] Registered new tserver with Master: 730d23ec728547609406d0fceaf73197 (127.28.112.1:43077)
W20260809 07:53:44.413514 7364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:44.413750 7364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:44.413791 7364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:44.417918 7364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:44.418004 7364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:44.418098 7364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:53:44.423246 7364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:42033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.2.keytab
--principal=oryx/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:39549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=34257
--tserver_master_addrs=127.28.112.62:37291,127.28.112.61:42839,127.28.112.60:41053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:44.424149 7364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:44.425108 7364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:44.434305 7371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:44.434346 7364 server_base.cc:1034] running on GCE node
W20260809 07:53:44.434299 7372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:44.434307 7374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:44.434862 7364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:44.435420 7364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:44.436573 7364 hybrid_clock.cc:648] HybridClock initialized: now 1786262024436553 us; error 28 us; skew 500 ppm
Aug 09 07:53:44 dist-test-slave-vpvm krb5kdc[5823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262024, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:44.439360 7364 init.cc:377] Logged in from keytab as oryx/127.28.112.2@KRBTEST.COM (short username oryx)
I20260809 07:53:44.440353 7364 webserver.cc:460] Webserver started at http://127.28.112.2:34257/ using document root <none> and password file <none>
I20260809 07:53:44.441010 7364 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:44.441102 7364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:44.444890 7364 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:44.446908 7387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:44.448062 7364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:44.448172 7364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0"
format_stamp: "Formatted at 2026-08-09 07:53:43 on dist-test-slave-vpvm"
server_key: "9c92d5167e9b26bb5abd9ec49c170685"
server_key_iv: "b6a5d6ae45642af676e52573e1358179"
server_key_version: "encryptionkey@0"
I20260809 07:53:44.448686 7364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:44.457429 6587 heartbeater.cc:503] Master 127.28.112.62:37291 was elected leader, sending a full tablet report...
I20260809 07:53:44.468019 7364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:44.470770 7364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:44.470932 7364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:44.471494 7364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:44.472314 7364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:44.472360 7364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:44.472411 7364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:44.472437 7364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:44.480778 7364 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:39549
I20260809 07:53:44.480831 7564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:39549 every 8 connection(s)
I20260809 07:53:44.481724 7364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:53:44.489832 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 7364
I20260809 07:53:44.490041 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 6393
Aug 09 07:53:44 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262024, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM
I20260809 07:53:44.501384 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.483437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:55693 (local address 127.28.112.62:37291)
0809 07:53:44.483696 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.483700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.484738 (+ 1038us) server_negotiation.cc:407] Connection header received
0809 07:53:44.485780 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.485784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.485842 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.485894 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:44.487268 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.488038 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.489057 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.489296 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.489392 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.498007 (+ 8615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:44.498011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:44.498013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:44.498040 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:44.499365 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.499891 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.499894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.499896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.499972 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.500307 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.500310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.500312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.500500 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:44.500558 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.501122 (+ 564us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.501243 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260809 07:53:44.503027 7570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.483938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.2:55693)
0809 07:53:44.484658 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.484671 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.485554 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.486108 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.486116 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.486527 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:44.487079 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.487098 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.488162 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.488165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.488934 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.488943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.489436 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.489440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.489645 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.496071 (+ 6426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:44.496090 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:44.497806 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:44.499493 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.499498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.499513 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.499791 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.500058 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.500061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.500062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.500213 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.500623 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:44.500629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:44.500884 (+ 255us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.501834 (+ 950us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.502169 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":159,"spinlock_wait_cycles":8448,"thread_start_us":124,"threads_started":1}
Aug 09 07:53:44 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262024, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.61@KRBTEST.COM
I20260809 07:53:44.507894 7572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.484937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:42839 (local address 127.28.112.2:53203)
0809 07:53:44.485161 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.485174 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.485558 (+ 384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.486107 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.486114 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.486524 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:44.487082 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.487097 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.488145 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.488148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.489145 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.489155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.489845 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.489849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.503239 (+ 13390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.503702 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:44.503717 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:44.504932 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:44.506510 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.506513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.506515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.506702 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.506964 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.506967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.506969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.507045 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.507408 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:44.507411 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:44.507498 (+ 87us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.507585 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.507719 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":101,"thread_start_us":85,"threads_started":1}
I20260809 07:53:44.507896 6977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.485071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:53203 (local address 127.28.112.61:42839)
0809 07:53:44.485189 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.485192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.485235 (+ 43us) server_negotiation.cc:407] Connection header received
0809 07:53:44.485760 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.485763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.485818 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.485888 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:44.487221 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.488025 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.489289 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.489603 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.489711 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.505074 (+ 15363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:44.505078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:44.505081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:44.505109 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:44.506402 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.506793 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.506797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.506798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.506879 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.507124 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.507128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.507129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.507317 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:44.507404 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.507654 (+ 250us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.507763 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 07:53:44.509097 7568 heartbeater.cc:347] Connected to a master server at 127.28.112.61:42839
I20260809 07:53:44.509332 7568 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:44.509817 7568 heartbeater.cc:511] Master 127.28.112.61:42839 requested a full tablet report, sending...
I20260809 07:53:44.510968 6799 ts_manager.cc:194] Registered new tserver with Master: 94d6943ad70f4d11a5e4f6f0a1244bc0 (127.28.112.2:39549)
I20260809 07:53:44.515007 7567 heartbeater.cc:347] Connected to a master server at 127.28.112.62:37291
I20260809 07:53:44.515123 7567 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:44.515372 7567 heartbeater.cc:511] Master 127.28.112.62:37291 requested a full tablet report, sending...
I20260809 07:53:44.516095 5902 ts_manager.cc:194] Re-registered known tserver with Master: 94d6943ad70f4d11a5e4f6f0a1244bc0 (127.28.112.2:39549)
I20260809 07:53:44.516834 5902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:55693
Aug 09 07:53:44 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262024, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.60@KRBTEST.COM
I20260809 07:53:44.521337 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.483437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:48399 (local address 127.28.112.60:41053)
0809 07:53:44.483589 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.483593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.484665 (+ 1072us) server_negotiation.cc:407] Connection header received
0809 07:53:44.485851 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.485855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.485903 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.485998 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:44.487242 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.487763 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.489261 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.489566 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.489666 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.518358 (+ 28692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:44.518362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:44.518365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:44.518396 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:44.519751 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.520157 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.520161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.520163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.520246 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.520499 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.520502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.520504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.520695 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:44.520786 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.521052 (+ 266us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.521181 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 07:53:44.521358 7571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.483938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:41053 (local address 127.28.112.2:48399)
0809 07:53:44.484510 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.484565 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.485557 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.486108 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.486116 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.486531 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:44.487079 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.487098 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.487909 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.487912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.489144 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.489155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.489790 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.489794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.489888 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.516149 (+ 26261us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:44.516161 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:44.518180 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:44.519862 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.519866 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.519868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.520058 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.520331 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.520334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.520336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.520415 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.520787 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:44.520791 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:44.520876 (+ 85us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.520977 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.521169 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":128,"spinlock_wait_cycles":96768,"thread_start_us":171,"threads_started":1}
I20260809 07:53:44.522042 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:44345
--local_ip_for_outbound_sockets=127.28.112.3
--tserver_master_addrs=127.28.112.62:37291
--webserver_port=34869
--webserver_interface=127.28.112.3
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.3.keytab
--principal=oryx/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:42033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.112.62:37291,127.28.112.61:42839
--tserver_master_addrs=127.28.112.62:37291,127.28.112.61:42839,127.28.112.60:41053 with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 07:53:44.523571 7566 heartbeater.cc:347] Connected to a master server at 127.28.112.60:41053
I20260809 07:53:44.523653 7566 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:44.523831 7566 heartbeater.cc:511] Master 127.28.112.60:41053 requested a full tablet report, sending...
I20260809 07:53:44.524573 7081 ts_manager.cc:194] Registered new tserver with Master: 94d6943ad70f4d11a5e4f6f0a1244bc0 (127.28.112.2:39549)
I20260809 07:53:44.613267 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.608936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.28.112.60:41053)
0809 07:53:44.609088 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.609093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.609177 (+ 84us) server_negotiation.cc:407] Connection header received
0809 07:53:44.609307 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.609310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.609377 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.609508 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:44.609788 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.610614 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.611865 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.612513 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.612618 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.612774 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.613085 (+ 311us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.613161 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 07:53:44.613293 7574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.608846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:41053 (local address 127.0.0.1:42086)
0809 07:53:44.609080 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.609094 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.609215 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.609510 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.609512 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.609523 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:44.609652 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.609660 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.610760 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.610765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.611713 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.611726 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.612845 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.612849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.612947 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.612973 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.613053 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.613137 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":65,"threads_started":1}
I20260809 07:53:44.614001 7094 raft_consensus.cc:1203] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260809 07:53:44.614284 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64801c974cf64a69844c72bf20ac5983. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } } }
I20260809 07:53:44.614421 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: This master's current role is: LEARNER
W20260809 07:53:44.634505 7573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:44.634742 7573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:44.634802 7573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:44.638721 7573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:44.638772 7573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:44.638844 7573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:53:44.643934 7573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:42033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.3.keytab
--principal=oryx/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:44345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=34869
--tserver_master_addrs=127.28.112.62:37291,127.28.112.61:42839,127.28.112.60:41053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:44.644865 7573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:44.645808 7573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:44.655032 7584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:44.655158 7573 server_base.cc:1034] running on GCE node
W20260809 07:53:44.655040 7582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:44.655040 7581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:44.655799 7573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:44.656409 7573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:44.657562 7573 hybrid_clock.cc:648] HybridClock initialized: now 1786262024657540 us; error 28 us; skew 500 ppm
Aug 09 07:53:44 dist-test-slave-vpvm krb5kdc[5823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262024, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:44.660116 7573 init.cc:377] Logged in from keytab as oryx/127.28.112.3@KRBTEST.COM (short username oryx)
I20260809 07:53:44.660919 7573 webserver.cc:460] Webserver started at http://127.28.112.3:34869/ using document root <none> and password file <none>
I20260809 07:53:44.661433 7573 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:44.661516 7573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:44.665150 7573 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:44.667124 7597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:44.668287 7573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:44.668411 7573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "433a721039ab44cdbf224fda5eacdfdc"
format_stamp: "Formatted at 2026-08-09 07:53:43 on dist-test-slave-vpvm"
server_key: "ac9f77f9eb758e04eecb4bcdabe19167"
server_key_iv: "9f2847e28f88dda0132131afd49a5fa6"
server_key_version: "encryptionkey@0"
I20260809 07:53:44.668926 7573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:44.678535 7573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:44.681144 7573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:44.681308 7573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:44.681843 7573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:44.682699 7573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:44.682749 7573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:44.682803 7573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:44.682832 7573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:44.691254 7573 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:44345
I20260809 07:53:44.691305 7762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:44345 every 8 connection(s)
I20260809 07:53:44.692243 7573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:53:44.698529 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 7573
Aug 09 07:53:44 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262024, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.61@KRBTEST.COM
Aug 09 07:53:44 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262024, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.60@KRBTEST.COM
Aug 09 07:53:44 dist-test-slave-vpvm krb5kdc[5823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262024, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM
I20260809 07:53:44.708629 6977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.695628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:51705 (local address 127.28.112.61:42839)
0809 07:53:44.695763 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.695766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.695957 (+ 191us) server_negotiation.cc:407] Connection header received
0809 07:53:44.696528 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.696532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.696588 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.696660 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:44.698343 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.698904 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.700474 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.700698 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.700789 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.704625 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:44.704629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:44.704632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:44.704658 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:44.706282 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.706900 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.706904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.706905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.707005 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.707404 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.707408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.707410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.707618 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:44.707689 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.708362 (+ 673us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.708473 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 07:53:44.710248 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.695509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:42839 (local address 127.28.112.3:51705)
0809 07:53:44.695862 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.695875 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.696288 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.696851 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.696878 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.697342 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:44.698163 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.698185 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.699041 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.699044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.700320 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.700332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.700838 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.700842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.701024 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.701626 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:44.701646 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:44.704474 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:44.706409 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.706415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.706431 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.706780 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.707112 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.707115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.707117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.707290 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.707773 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:44.707779 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:44.708065 (+ 286us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.708884 (+ 819us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.709213 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":90,"spinlock_wait_cycles":43520,"thread_start_us":79,"threads_started":1}
I20260809 07:53:44.710518 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.694435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.3:37305)
0809 07:53:44.695224 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.695280 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.696283 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.696851 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.696860 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.697350 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:44.698170 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.698185 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.701052 (+ 2867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.701056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.701802 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.701813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.702524 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.702528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.702608 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.703027 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:44.703042 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:44.704406 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:44.707702 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.707705 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.707706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.707915 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.708229 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.708232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.708234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.708563 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.709348 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:44.709351 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:44.709441 (+ 90us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.709531 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.709653 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":151,"spinlock_wait_cycles":65536,"thread_start_us":157,"threads_started":1}
I20260809 07:53:44.711697 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.694435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:41053 (local address 127.28.112.3:41303)
0809 07:53:44.695224 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.695280 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.696283 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.696851 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.696860 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.697345 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:44.698166 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.698185 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.699917 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.699920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.700810 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.700820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.701623 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.701627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.701739 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.702157 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:44.702171 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:44.704045 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:44.706610 (+ 2565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.706613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.706615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.707617 (+ 1002us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.708852 (+ 1235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:44.708855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:44.708857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:44.708945 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:44.711223 (+ 2278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:44.711226 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:44.711306 (+ 80us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.711390 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.711510 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":135,"spinlock_wait_cycles":16768,"thread_start_us":166,"threads_started":1}
I20260809 07:53:44.711841 7766 heartbeater.cc:347] Connected to a master server at 127.28.112.61:42839
I20260809 07:53:44.712076 7766 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:44.712636 7766 heartbeater.cc:511] Master 127.28.112.61:42839 requested a full tablet report, sending...
I20260809 07:53:44.713779 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.693941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:37305 (local address 127.28.112.62:37291)
0809 07:53:44.694085 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.694090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.695386 (+ 1296us) server_negotiation.cc:407] Connection header received
0809 07:53:44.696596 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.696599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.696652 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.696724 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:44.700095 (+ 3371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.700857 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.701970 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.702284 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.702382 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.704555 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:44.704560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:44.704563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:44.704590 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:44.707363 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.708033 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.708037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.708038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.708127 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.709031 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.709034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.709036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.709222 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:44.709304 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.713525 (+ 4221us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.713634 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 07:53:44.713819 6799 ts_manager.cc:194] Registered new tserver with Master: 433a721039ab44cdbf224fda5eacdfdc (127.28.112.3:44345)
I20260809 07:53:44.714262 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.693943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:41303 (local address 127.28.112.60:41053)
0809 07:53:44.694085 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.694090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.695386 (+ 1296us) server_negotiation.cc:407] Connection header received
0809 07:53:44.696528 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.696532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.696590 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.696679 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:44.699038 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.699795 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.700913 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.701217 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.701316 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.704230 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:44.704234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:44.704236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:44.704263 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:44.706491 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.708678 (+ 2187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.708681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.708683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.708765 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.710878 (+ 2113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.710881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.710883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.711104 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:44.711183 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.714012 (+ 2829us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.714121 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 07:53:44.714322 7765 heartbeater.cc:347] Connected to a master server at 127.28.112.62:37291
I20260809 07:53:44.714408 7765 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:44.714568 7765 heartbeater.cc:511] Master 127.28.112.62:37291 requested a full tablet report, sending...
I20260809 07:53:44.714898 7763 heartbeater.cc:347] Connected to a master server at 127.28.112.60:41053
I20260809 07:53:44.714980 7763 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:44.715005 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.701452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.28.112.62:37291)
0809 07:53:44.702721 (+ 1269us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.702725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.703004 (+ 279us) server_negotiation.cc:407] Connection header received
0809 07:53:44.703053 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.703056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.703111 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.703213 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:44.705565 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.706367 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.707449 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.707780 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.707867 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.709803 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:44.709807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:44.709809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:44.709833 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:44.711851 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.712256 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.712259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.712261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.712340 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.712789 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.712793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.712794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.712954 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:44.713074 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.713308 (+ 234us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.713420 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":65,"threads_started":1}
I20260809 07:53:44.715144 7763 heartbeater.cc:511] Master 127.28.112.60:41053 requested a full tablet report, sending...
I20260809 07:53:44.715202 5902 ts_manager.cc:194] Re-registered known tserver with Master: 433a721039ab44cdbf224fda5eacdfdc (127.28.112.3:44345)
I20260809 07:53:44.715794 7081 ts_manager.cc:194] Registered new tserver with Master: 433a721039ab44cdbf224fda5eacdfdc (127.28.112.3:44345)
I20260809 07:53:44.716024 5902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:37305
I20260809 07:53:44.718343 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.702601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42088 (local address 127.28.112.60:41053)
0809 07:53:44.703184 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.703188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.703698 (+ 510us) server_negotiation.cc:407] Connection header received
0809 07:53:44.703734 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.703736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.703776 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.703846 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:44.704936 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.705687 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.709414 (+ 3727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.709757 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.709830 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.712487 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:44.712491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:44.712493 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:44.712516 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:44.714594 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.716793 (+ 2199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.716797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.716799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.716885 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.717196 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.717200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.717202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.717471 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:44.717596 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.718078 (+ 482us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.718196 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":85,"threads_started":1}
I20260809 07:53:44.718477 5902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:53:44.721364 5902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:53:44.723493 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.701690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49158 (local address 127.28.112.61:42839)
0809 07:53:44.703314 (+ 1624us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.703317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.703330 (+ 13us) server_negotiation.cc:407] Connection header received
0809 07:53:44.703713 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.703717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.703761 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.703835 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:44.706614 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.707379 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.711112 (+ 3733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.711433 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.711518 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.716025 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:44.716029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:44.716032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:44.716056 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:44.718008 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.722330 (+ 4322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.722334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.722335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.722417 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:44.722676 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:44.722679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:44.722681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:44.722858 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:44.722973 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.723211 (+ 238us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.723332 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1560,"thread_start_us":75,"threads_started":1}
I20260809 07:53:44.723709 7025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: NON_VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: true } } }
I20260809 07:53:44.723830 7025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:44.725910 7025 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983: attempting to promote NON_VOTER add5262818464f139ffb4c5bade4a733 to VOTER
I20260809 07:53:44.726230 7025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } }
I20260809 07:53:44.727273 6812 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 1 FOLLOWER]: Refusing update from remote peer 64801c974cf64a69844c72bf20ac5983: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260809 07:53:44.727313 7094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 1 LEARNER]: Refusing update from remote peer 64801c974cf64a69844c72bf20ac5983: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260809 07:53:44.727603 7788 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 07:53:44.727743 7025 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [LEADER]: Connected to new peer: Peer: permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 07:53:44.729316 7025 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, add5262818464f139ffb4c5bade4a733 (127.28.112.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } } }
I20260809 07:53:44.729770 6812 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, add5262818464f139ffb4c5bade4a733 (127.28.112.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } } }
I20260809 07:53:44.730108 7094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, add5262818464f139ffb4c5bade4a733 (127.28.112.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } } }
I20260809 07:53:44.731118 7025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } } }
I20260809 07:53:44.731257 7025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:44.731428 7025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } } }
I20260809 07:53:44.731554 7025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:44.731519 7788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } } }
I20260809 07:53:44.731781 7788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:44.731948 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } } }
I20260809 07:53:44.732038 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:53:44.731997 7798 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64801c974cf64a69844c72bf20ac5983" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } } }
I20260809 07:53:44.732103 7798 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:53:44.742081 7574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.736737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:39549 (local address 127.0.0.1:51202)
0809 07:53:44.736869 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.736884 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.736958 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.737735 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.737738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.737752 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:44.737836 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.737841 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.739303 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.739306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.740395 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.740406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.741674 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.741678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.741763 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.741784 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.741870 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.741945 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 07:53:44.742349 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.736937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51202 (local address 127.28.112.2:39549)
0809 07:53:44.737236 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.737249 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.737272 (+ 23us) server_negotiation.cc:407] Connection header received
0809 07:53:44.737350 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.737356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.737585 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.738007 (+ 422us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:44.738149 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.739201 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.740525 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.741559 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.741668 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.741814 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.741902 (+ 88us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.742019 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":102,"threads_started":1}
I20260809 07:53:44.743386 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.737717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:43077 (local address 127.0.0.1:40626)
0809 07:53:44.737945 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.737987 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.738059 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.739323 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.739327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.739346 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:44.739433 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.739439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.740754 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.740757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.742130 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.742141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.742979 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.742984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.743069 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.743095 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.743187 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.743255 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":72,"thread_start_us":79,"threads_started":1}
I20260809 07:53:44.743443 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.738383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40626 (local address 127.28.112.1:43077)
0809 07:53:44.738702 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.738718 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.738740 (+ 22us) server_negotiation.cc:407] Connection header received
0809 07:53:44.738817 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.738822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.739013 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.739161 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:44.739548 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.740614 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.742265 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.742864 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.742953 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.743076 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.743214 (+ 138us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.743301 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":112,"threads_started":1}
I20260809 07:53:44.744372 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.737325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:44345 (local address 127.0.0.1:53686)
0809 07:53:44.737554 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.737567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.737647 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.739703 (+ 2056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.739706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.739724 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:44.739813 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.739819 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.740749 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.740752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.742130 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.742141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.744003 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.744007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.744094 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.744117 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.744188 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.744248 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":80,"thread_start_us":102,"threads_started":1}
I20260809 07:53:44.745122 7460 tablet_service.cc:1467] Processing CreateTablet for tablet e3d95f9c65f8439c958d7ffb9a71833f (DEFAULT_TABLE table=test-table [id=ddce20dcb2eb4d1eac16064cee831f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:44.745697 7247 tablet_service.cc:1467] Processing CreateTablet for tablet e3d95f9c65f8439c958d7ffb9a71833f (DEFAULT_TABLE table=test-table [id=ddce20dcb2eb4d1eac16064cee831f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:44.746268 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.737474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53686 (local address 127.28.112.3:44345)
0809 07:53:44.739318 (+ 1844us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.739329 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.739348 (+ 19us) server_negotiation.cc:407] Connection header received
0809 07:53:44.739415 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.739420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.739589 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.739713 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:44.739900 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.740652 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.742377 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.743878 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.743986 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.744111 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.746041 (+ 1930us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.746136 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1748,"thread_start_us":81,"threads_started":1}
I20260809 07:53:44.746702 7247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d95f9c65f8439c958d7ffb9a71833f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:44.748634 7670 tablet_service.cc:1467] Processing CreateTablet for tablet e3d95f9c65f8439c958d7ffb9a71833f (DEFAULT_TABLE table=test-table [id=ddce20dcb2eb4d1eac16064cee831f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:44.749522 7670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d95f9c65f8439c958d7ffb9a71833f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:44.750262 7460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d95f9c65f8439c958d7ffb9a71833f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:44.756672 7806 tablet_bootstrap.cc:492] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc: Bootstrap starting.
I20260809 07:53:44.758340 7804 tablet_bootstrap.cc:492] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197: Bootstrap starting.
I20260809 07:53:44.759020 7806 tablet_bootstrap.cc:654] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:44.759879 7806 log.cc:824] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:44.760398 7804 tablet_bootstrap.cc:654] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:44.761165 7804 log.cc:824] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:44.761850 7810 tablet_bootstrap.cc:492] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0: Bootstrap starting.
I20260809 07:53:44.762279 7806 tablet_bootstrap.cc:492] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc: No bootstrap required, opened a new log
I20260809 07:53:44.762514 7806 ts_tablet_manager.cc:1397] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 07:53:44.763121 7804 tablet_bootstrap.cc:492] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197: No bootstrap required, opened a new log
I20260809 07:53:44.763304 7804 ts_tablet_manager.cc:1397] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 07:53:44.763861 7810 tablet_bootstrap.cc:654] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:44.764652 7810 log.cc:824] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:44.766395 7806 raft_consensus.cc:348] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } } peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } }
I20260809 07:53:44.766664 7806 raft_consensus.cc:374] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:44.766701 7810 tablet_bootstrap.cc:492] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0: No bootstrap required, opened a new log
I20260809 07:53:44.766727 7806 raft_consensus.cc:729] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433a721039ab44cdbf224fda5eacdfdc, State: Initialized, Role: FOLLOWER
I20260809 07:53:44.766870 7810 ts_tablet_manager.cc:1397] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 07:53:44.766983 7804 raft_consensus.cc:348] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } } peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } }
I20260809 07:53:44.767174 7804 raft_consensus.cc:374] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:44.767226 7804 raft_consensus.cc:729] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d23ec728547609406d0fceaf73197, State: Initialized, Role: FOLLOWER
I20260809 07:53:44.767225 7806 consensus_queue.cc:260] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } } peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } }
I20260809 07:53:44.768070 7804 consensus_queue.cc:260] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } } peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } }
I20260809 07:53:44.768249 7806 ts_tablet_manager.cc:1428] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 07:53:44.768635 7350 heartbeater.cc:503] Master 127.28.112.62:37291 was elected leader, sending a full tablet report...
I20260809 07:53:44.771147 7765 heartbeater.cc:503] Master 127.28.112.62:37291 was elected leader, sending a full tablet report...
I20260809 07:53:44.771015 7810 raft_consensus.cc:348] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } } peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } }
I20260809 07:53:44.771245 7810 raft_consensus.cc:374] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:44.771294 7810 raft_consensus.cc:729] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d6943ad70f4d11a5e4f6f0a1244bc0, State: Initialized, Role: FOLLOWER
I20260809 07:53:44.771695 7810 consensus_queue.cc:260] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } } peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } }
I20260809 07:53:44.772586 7810 ts_tablet_manager.cc:1428] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 07:53:44.773017 7567 heartbeater.cc:503] Master 127.28.112.62:37291 was elected leader, sending a full tablet report...
I20260809 07:53:44.773567 7804 ts_tablet_manager.cc:1428] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
W20260809 07:53:44.778437 7352 tablet.cc:2369] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 07:53:44.944162 7767 tablet.cc:2369] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:53:44.959764 7815 raft_consensus.cc:482] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:44.959969 7815 raft_consensus.cc:504] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } } peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } }
I20260809 07:53:44.961139 7815 leader_election.cc:288] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d23ec728547609406d0fceaf73197 (127.28.112.1:43077), 94d6943ad70f4d11a5e4f6f0a1244bc0 (127.28.112.2:39549)
I20260809 07:53:44.964798 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.961505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:40311 (local address 127.28.112.1:43077)
0809 07:53:44.961666 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.961670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.961692 (+ 22us) server_negotiation.cc:407] Connection header received
0809 07:53:44.961858 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.961861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.961907 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.961999 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:44.962253 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.962732 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.963857 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.964246 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.964353 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.964404 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.964604 (+ 200us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.964673 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 07:53:44.964802 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.961512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:33105 (local address 127.28.112.2:39549)
0809 07:53:44.961635 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:44.961640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:44.961731 (+ 91us) server_negotiation.cc:407] Connection header received
0809 07:53:44.961858 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:44.961860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:44.961900 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:44.961985 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:44.962237 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.962731 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.963860 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.964236 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.964335 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.964389 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:44.964624 (+ 235us) server_negotiation.cc:299] Negotiation successful
0809 07:53:44.964685 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 07:53:44.964825 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.961402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:43077 (local address 127.28.112.3:40311)
0809 07:53:44.961633 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.961648 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.961745 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.962013 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.962016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.962033 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:44.962150 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.962155 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.962840 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.962843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.963737 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.963746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.964353 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.964357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.964457 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.964502 (+ 45us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.964568 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.964639 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":46}
I20260809 07:53:44.964824 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:44.961408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:39549 (local address 127.28.112.3:33105)
0809 07:53:44.961651 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:44.961665 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:44.961761 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:44.962007 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:44.962010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:44.962025 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:44.962139 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.962145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.962836 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.962839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.963741 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:44.963751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:44.964381 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:44.964385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:44.964508 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:44.964527 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 07:53:44.964598 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 07:53:44.964671 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":143}
I20260809 07:53:44.965533 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d95f9c65f8439c958d7ffb9a71833f" candidate_uuid: "433a721039ab44cdbf224fda5eacdfdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d23ec728547609406d0fceaf73197" is_pre_election: true
I20260809 07:53:44.965617 7494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d95f9c65f8439c958d7ffb9a71833f" candidate_uuid: "433a721039ab44cdbf224fda5eacdfdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" is_pre_election: true
I20260809 07:53:44.965847 7273 raft_consensus.cc:2454] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 433a721039ab44cdbf224fda5eacdfdc in term 0.
I20260809 07:53:44.965910 7494 raft_consensus.cc:2454] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 433a721039ab44cdbf224fda5eacdfdc in term 0.
I20260809 07:53:44.966401 7598 leader_election.cc:302] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433a721039ab44cdbf224fda5eacdfdc, 730d23ec728547609406d0fceaf73197; no voters:
I20260809 07:53:44.966676 7815 raft_consensus.cc:2781] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:53:44.966811 7815 raft_consensus.cc:482] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:53:44.966861 7815 raft_consensus.cc:3037] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:44.967841 7815 raft_consensus.cc:504] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } } peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } }
I20260809 07:53:44.968163 7815 leader_election.cc:288] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [CANDIDATE]: Term 1 election: Requested vote from peers 730d23ec728547609406d0fceaf73197 (127.28.112.1:43077), 94d6943ad70f4d11a5e4f6f0a1244bc0 (127.28.112.2:39549)
I20260809 07:53:44.968497 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d95f9c65f8439c958d7ffb9a71833f" candidate_uuid: "433a721039ab44cdbf224fda5eacdfdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d23ec728547609406d0fceaf73197"
I20260809 07:53:44.968487 7494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d95f9c65f8439c958d7ffb9a71833f" candidate_uuid: "433a721039ab44cdbf224fda5eacdfdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0"
I20260809 07:53:44.968612 7494 raft_consensus.cc:3037] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:44.968612 7273 raft_consensus.cc:3037] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:44.969823 7273 raft_consensus.cc:2454] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 433a721039ab44cdbf224fda5eacdfdc in term 1.
I20260809 07:53:44.969823 7494 raft_consensus.cc:2454] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 433a721039ab44cdbf224fda5eacdfdc in term 1.
I20260809 07:53:44.970103 7604 leader_election.cc:302] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433a721039ab44cdbf224fda5eacdfdc, 94d6943ad70f4d11a5e4f6f0a1244bc0; no voters:
I20260809 07:53:44.970273 7815 raft_consensus.cc:2781] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:44.970504 7815 raft_consensus.cc:686] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [term 1 LEADER]: Becoming Leader. State: Replica: 433a721039ab44cdbf224fda5eacdfdc, State: Running, Role: LEADER
I20260809 07:53:44.970827 7815 consensus_queue.cc:237] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } } peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } }
I20260809 07:53:44.974148 5901 catalog_manager.cc:5495] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc reported cstate change: term changed from 0 to 1, leader changed from <none> to 433a721039ab44cdbf224fda5eacdfdc (127.28.112.3). New cstate: current_term: 1 leader_uuid: "433a721039ab44cdbf224fda5eacdfdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } health_report { overall_health: HEALTHY } } }
W20260809 07:53:44.983582 7569 tablet.cc:2369] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:53:45.042441 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:45.038904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53692 (local address 127.28.112.3:44345)
0809 07:53:45.039033 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:45.039038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:45.039052 (+ 14us) server_negotiation.cc:407] Connection header received
0809 07:53:45.039139 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:45.039142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:45.039199 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:45.039293 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:45.039542 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:45.040060 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:45.040902 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:45.041154 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:45.041246 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:45.041561 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:45.042036 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:45.042143 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:45.042243 (+ 100us) server_negotiation.cc:299] Negotiation successful
0809 07:53:45.042304 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 07:53:45.048030 7494 raft_consensus.cc:1261] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0 [term 1 FOLLOWER]: Refusing update from remote peer 433a721039ab44cdbf224fda5eacdfdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:45.048028 7273 raft_consensus.cc:1261] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197 [term 1 FOLLOWER]: Refusing update from remote peer 433a721039ab44cdbf224fda5eacdfdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:45.048695 7815 consensus_queue.cc:1035] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:45.048884 7822 consensus_queue.cc:1035] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:45.057163 7827 mvcc.cc:205] Tried to move back new op lower bound from 7316529254590115840 to 7316529254282829824. Current Snapshot: MvccSnapshot[applied={T|T < 7316529254590115840}]
I20260809 07:53:45.057370 7829 mvcc.cc:205] Tried to move back new op lower bound from 7316529254590115840 to 7316529254282829824. Current Snapshot: MvccSnapshot[applied={T|T < 7316529254590115840}]
I20260809 07:53:45.066938 5900 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:45.067097 5900 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:45.071151 5900 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64801c974cf64a69844c72bf20ac5983: Sending DeleteTablet for 3 replicas of tablet e3d95f9c65f8439c958d7ffb9a71833f
I20260809 07:53:45.072396 7460 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d95f9c65f8439c958d7ffb9a71833f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:45 UTC) from {username='oryx', principal='oryx/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:51202
I20260809 07:53:45.072443 7247 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d95f9c65f8439c958d7ffb9a71833f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:45 UTC) from {username='oryx', principal='oryx/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:40626
I20260809 07:53:45.073293 7843 tablet_replica.cc:330] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197: stopping tablet replica
I20260809 07:53:45.073676 7843 raft_consensus.cc:2229] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:45.073686 7670 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d95f9c65f8439c958d7ffb9a71833f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:45 UTC) from {username='oryx', principal='oryx/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:53686
I20260809 07:53:45.074138 7843 raft_consensus.cc:2258] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:45.074417 7847 tablet_replica.cc:330] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc: stopping tablet replica
I20260809 07:53:45.074699 7847 raft_consensus.cc:2229] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:53:45.074959 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 5842
I20260809 07:53:45.075031 7847 raft_consensus.cc:2258] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:45.080763 7842 tablet_replica.cc:330] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0: stopping tablet replica
I20260809 07:53:45.081147 7842 raft_consensus.cc:2229] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:45.081589 7842 raft_consensus.cc:2258] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:45.081857 7843 ts_tablet_manager.cc:1905] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:45.083237 7847 ts_tablet_manager.cc:1905] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:45.084843 7843 ts_tablet_manager.cc:1918] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:53:45.085006 7843 log.cc:1196] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-0/wal/wals/e3d95f9c65f8439c958d7ffb9a71833f
I20260809 07:53:45.085376 7843 ts_tablet_manager.cc:1939] T e3d95f9c65f8439c958d7ffb9a71833f P 730d23ec728547609406d0fceaf73197: Deleting consensus metadata
I20260809 07:53:45.085902 7847 ts_tablet_manager.cc:1918] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:53:45.086023 7847 log.cc:1196] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/e3d95f9c65f8439c958d7ffb9a71833f
I20260809 07:53:45.086329 7847 ts_tablet_manager.cc:1939] T e3d95f9c65f8439c958d7ffb9a71833f P 433a721039ab44cdbf224fda5eacdfdc: Deleting consensus metadata
I20260809 07:53:45.088670 7842 ts_tablet_manager.cc:1905] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 07:53:45.089923 7176 connection.cc:535] server connection from 127.0.0.1:40626 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 07:53:45.089938 7604 connection.cc:535] server connection from 127.0.0.1:53686 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 07:53:45.091233 7842 ts_tablet_manager.cc:1918] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:53:45.091321 7842 log.cc:1196] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/e3d95f9c65f8439c958d7ffb9a71833f
I20260809 07:53:45.091626 7842 ts_tablet_manager.cc:1939] T e3d95f9c65f8439c958d7ffb9a71833f P 94d6943ad70f4d11a5e4f6f0a1244bc0: Deleting consensus metadata
W20260809 07:53:45.092401 7394 connection.cc:441] server connection from 127.0.0.1:51202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51202 (request call id 1) could send its response
I20260809 07:53:45.102641 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:45.073388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49160 (local address 127.28.112.61:42839)
0809 07:53:45.073513 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:45.073518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:45.087715 (+ 14197us) server_negotiation.cc:407] Connection header received
0809 07:53:45.092616 (+ 4901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:45.092619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:45.092674 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:45.092744 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:45.093480 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:45.094244 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:45.095329 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:45.095680 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:45.095765 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:45.097634 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:45.097638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:45.097640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:45.097669 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:45.099843 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:45.100976 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:45.100981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:45.100983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:45.101077 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:45.101641 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:45.101645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:45.101647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:45.101859 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:45.101992 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:45.102350 (+ 358us) server_negotiation.cc:299] Negotiation successful
0809 07:53:45.102511 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":50}
I20260809 07:53:45.103541 6977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:45.093618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.28.112.61:42839)
0809 07:53:45.093724 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:45.093728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:45.094702 (+ 974us) server_negotiation.cc:407] Connection header received
0809 07:53:45.094871 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:45.094875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:45.094926 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:45.095006 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:45.095930 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:45.096680 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:45.097811 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:45.098187 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:45.098257 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:45.099712 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:45.099716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:45.099719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:45.099762 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:45.101257 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:45.101799 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:45.101803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:45.101804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:45.101890 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:45.102224 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:45.102227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:45.102229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:45.102890 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:45.103002 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:45.103279 (+ 277us) server_negotiation.cc:299] Negotiation successful
0809 07:53:45.103389 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":427,"server-negotiator.queue_time_us":41}
I20260809 07:53:45.105139 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:45.094300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42110 (local address 127.28.112.60:41053)
0809 07:53:45.094413 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:45.094417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:45.095026 (+ 609us) server_negotiation.cc:407] Connection header received
0809 07:53:45.095177 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:45.095180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:45.095230 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:45.095281 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:45.096514 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:45.097300 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:45.098506 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:45.098860 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:45.098952 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:45.100506 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:45.100510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:45.100513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:45.100541 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:45.102048 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:45.102478 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:45.102482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:45.102484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:45.102566 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:45.102867 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:45.102871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:45.102873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:45.103098 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:45.103211 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:45.104886 (+ 1675us) server_negotiation.cc:299] Negotiation successful
0809 07:53:45.105001 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 07:53:45.110330 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:45.073823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42100 (local address 127.28.112.60:41053)
0809 07:53:45.074092 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:45.074096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:45.103389 (+ 29293us) server_negotiation.cc:407] Connection header received
0809 07:53:45.103565 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:45.103568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:45.103613 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:45.103668 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:45.104373 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:45.105059 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:45.106158 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:45.106464 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:45.106550 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:45.107712 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:45.107716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:45.107719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:45.107743 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:45.108885 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:45.109248 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:45.109252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:45.109254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:45.109328 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:45.109575 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:45.109578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:45.109580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:45.109741 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:45.109846 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:45.110083 (+ 237us) server_negotiation.cc:299] Negotiation successful
0809 07:53:45.110191 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260809 07:53:46.061583 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:46.061117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.1:34775)
0809 07:53:46.061374 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:46.061466 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":82,"threads_started":1}
I20260809 07:53:46.061856 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:46.061430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.3:60827)
0809 07:53:46.061670 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:46.061763 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
W20260809 07:53:46.061897 7350 heartbeater.cc:644] Failed to heartbeat to 127.28.112.62:37291 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
W20260809 07:53:46.062127 7765 heartbeater.cc:644] Failed to heartbeat to 127.28.112.62:37291 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
I20260809 07:53:46.064182 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:46.063539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.2:40549)
0809 07:53:46.064006 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:46.064092 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":65,"threads_started":1}
W20260809 07:53:46.064471 7567 heartbeater.cc:644] Failed to heartbeat to 127.28.112.62:37291 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
I20260809 07:53:46.572715 7862 raft_consensus.cc:482] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 64801c974cf64a69844c72bf20ac5983)
I20260809 07:53:46.572815 7863 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 64801c974cf64a69844c72bf20ac5983)
I20260809 07:53:46.572914 7862 raft_consensus.cc:504] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } }
I20260809 07:53:46.573033 7863 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } }
I20260809 07:53:46.574033 7863 leader_election.cc:288] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 64801c974cf64a69844c72bf20ac5983 (127.28.112.62:37291), add5262818464f139ffb4c5bade4a733 (127.28.112.60:41053)
I20260809 07:53:46.574393 7862 leader_election.cc:288] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 64801c974cf64a69844c72bf20ac5983 (127.28.112.62:37291), bbbca9ad28064f5bab856505d7dff710 (127.28.112.61:42839)
I20260809 07:53:46.574851 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:46.574177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:50016)
0809 07:53:46.574660 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:46.574745 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":54,"threads_started":1}
W20260809 07:53:46.575049 6780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
I20260809 07:53:46.575301 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:46.574802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:50032)
0809 07:53:46.575123 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:46.575206 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":77,"threads_started":1}
W20260809 07:53:46.575548 7063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
I20260809 07:53:46.576119 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:46.575821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:50036)
0809 07:53:46.575981 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:46.576035 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260809 07:53:46.576261 6780 leader_election.cc:334] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 64801c974cf64a69844c72bf20ac5983 (127.28.112.62:37291): Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
I20260809 07:53:46.576503 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:46.576234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:50042)
0809 07:53:46.576383 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:46.576442 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260809 07:53:46.576599 7063 leader_election.cc:334] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 64801c974cf64a69844c72bf20ac5983 (127.28.112.62:37291): Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
I20260809 07:53:46.579028 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:46.574210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:42839 (local address 127.0.0.1:49180)
0809 07:53:46.574481 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:46.574500 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:46.574618 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:46.575690 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:46.575694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:46.575714 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:46.575818 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:46.575825 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:46.576878 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:46.576880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:46.578042 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:46.578054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:46.578706 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:46.578708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:46.578786 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:46.578809 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 07:53:46.578864 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 07:53:46.578927 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":65,"threads_started":1}
I20260809 07:53:46.579078 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:46.574447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49180 (local address 127.28.112.61:42839)
0809 07:53:46.575466 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:46.575470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:46.575482 (+ 12us) server_negotiation.cc:407] Connection header received
0809 07:53:46.575519 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:46.575522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:46.575559 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:46.575930 (+ 371us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:46.576058 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:46.576781 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:46.578157 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:46.578597 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:46.578704 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:46.578763 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:46.578890 (+ 127us) server_negotiation.cc:299] Negotiation successful
0809 07:53:46.578952 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":958,"thread_start_us":145,"threads_started":1}
I20260809 07:53:46.579627 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:46.574209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:41053 (local address 127.0.0.1:42120)
0809 07:53:46.574767 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:46.574779 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:46.574853 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:46.575284 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:46.575287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:46.575307 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:46.576043 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:46.576051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:46.577294 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:46.577298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:46.578395 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:46.578405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:46.579280 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:46.579284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:46.579371 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:46.579393 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 07:53:46.579467 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 07:53:46.579538 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":18,"thread_start_us":72,"threads_started":1}
I20260809 07:53:46.579586 6812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "add5262818464f139ffb4c5bade4a733" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bbbca9ad28064f5bab856505d7dff710" is_pre_election: true
I20260809 07:53:46.579689 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:46.574568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42120 (local address 127.28.112.60:41053)
0809 07:53:46.575075 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:46.575078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:46.575090 (+ 12us) server_negotiation.cc:407] Connection header received
0809 07:53:46.575127 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:46.575129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:46.575186 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:46.575271 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:46.576420 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:46.577185 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:46.578503 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:46.579061 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:46.579163 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:46.579218 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:46.579497 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 07:53:46.579571 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":76,"threads_started":1}
I20260809 07:53:46.579798 6812 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate add5262818464f139ffb4c5bade4a733 in term 1.
I20260809 07:53:46.580096 7094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbbca9ad28064f5bab856505d7dff710" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "add5262818464f139ffb4c5bade4a733" is_pre_election: true
I20260809 07:53:46.580132 7059 leader_election.cc:302] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: add5262818464f139ffb4c5bade4a733, bbbca9ad28064f5bab856505d7dff710; no voters: 64801c974cf64a69844c72bf20ac5983
I20260809 07:53:46.580281 7094 raft_consensus.cc:2454] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbbca9ad28064f5bab856505d7dff710 in term 1.
I20260809 07:53:46.580365 7862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 07:53:46.580435 7862 raft_consensus.cc:482] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 64801c974cf64a69844c72bf20ac5983)
I20260809 07:53:46.580473 7862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 1 FOLLOWER]: Advancing to term 2
I20260809 07:53:46.580575 6775 leader_election.cc:302] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: add5262818464f139ffb4c5bade4a733, bbbca9ad28064f5bab856505d7dff710; no voters: 64801c974cf64a69844c72bf20ac5983
I20260809 07:53:46.580761 7863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 07:53:46.580819 7863 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 64801c974cf64a69844c72bf20ac5983)
I20260809 07:53:46.580864 7863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 1 FOLLOWER]: Advancing to term 2
I20260809 07:53:46.581565 7862 raft_consensus.cc:504] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } }
I20260809 07:53:46.581779 7863 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } }
I20260809 07:53:46.581889 7862 leader_election.cc:288] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [CANDIDATE]: Term 2 election: Requested vote from peers 64801c974cf64a69844c72bf20ac5983 (127.28.112.62:37291), bbbca9ad28064f5bab856505d7dff710 (127.28.112.61:42839)
I20260809 07:53:46.582201 7863 leader_election.cc:288] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [CANDIDATE]: Term 2 election: Requested vote from peers 64801c974cf64a69844c72bf20ac5983 (127.28.112.62:37291), add5262818464f139ffb4c5bade4a733 (127.28.112.60:41053)
I20260809 07:53:46.582283 6812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "add5262818464f139ffb4c5bade4a733" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bbbca9ad28064f5bab856505d7dff710"
I20260809 07:53:46.582392 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:46.582011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:50052)
0809 07:53:46.582109 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:46.582160 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260809 07:53:46.582446 6812 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate add5262818464f139ffb4c5bade4a733 in current term 2: Already voted for candidate bbbca9ad28064f5bab856505d7dff710 in this term.
I20260809 07:53:46.582495 7094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbbca9ad28064f5bab856505d7dff710" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "add5262818464f139ffb4c5bade4a733"
I20260809 07:53:46.582629 7094 raft_consensus.cc:2379] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bbbca9ad28064f5bab856505d7dff710 in current term 2: Already voted for candidate add5262818464f139ffb4c5bade4a733 in this term.
I20260809 07:53:46.582952 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:46.582230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:50056)
0809 07:53:46.582620 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:46.582890 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208}
I20260809 07:53:46.582980 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:46.582768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:50058)
0809 07:53:46.582888 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:46.582930 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260809 07:53:46.583170 7063 leader_election.cc:334] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 64801c974cf64a69844c72bf20ac5983 (127.28.112.62:37291): Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
I20260809 07:53:46.583228 7063 leader_election.cc:302] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: add5262818464f139ffb4c5bade4a733; no voters: 64801c974cf64a69844c72bf20ac5983, bbbca9ad28064f5bab856505d7dff710
I20260809 07:53:46.583361 7862 raft_consensus.cc:2726] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 07:53:46.583544 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:46.583346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:50070)
0809 07:53:46.583437 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:46.583488 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260809 07:53:46.583643 6780 leader_election.cc:334] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 64801c974cf64a69844c72bf20ac5983 (127.28.112.62:37291): Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
I20260809 07:53:46.583709 6780 leader_election.cc:302] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbbca9ad28064f5bab856505d7dff710; no voters: 64801c974cf64a69844c72bf20ac5983, add5262818464f139ffb4c5bade4a733
I20260809 07:53:46.583832 7863 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 07:53:47.062913 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:47.062443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.1:53423)
0809 07:53:47.062760 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:47.062824 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":95,"threads_started":1}
I20260809 07:53:47.062968 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:47.062568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.3:57517)
0809 07:53:47.062830 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:47.062884 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":95,"threads_started":1}
I20260809 07:53:47.065407 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:47.064922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.2:52095)
0809 07:53:47.065282 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:47.065331 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":73,"threads_started":1}
I20260809 07:53:48.064072 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.063657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.1:39217)
0809 07:53:48.063919 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.063981 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":94,"threads_started":1}
I20260809 07:53:48.064122 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.063657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.3:50935)
0809 07:53:48.063979 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.064037 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":96,"threads_started":1}
I20260809 07:53:48.066468 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.066023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.2:35581)
0809 07:53:48.066317 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.066375 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":74,"threads_started":1}
I20260809 07:53:48.462076 7890 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:48.462224 7890 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } }
I20260809 07:53:48.462671 7890 leader_election.cc:288] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 64801c974cf64a69844c72bf20ac5983 (127.28.112.62:37291), add5262818464f139ffb4c5bade4a733 (127.28.112.60:41053)
I20260809 07:53:48.463160 7094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbbca9ad28064f5bab856505d7dff710" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "add5262818464f139ffb4c5bade4a733" is_pre_election: true
I20260809 07:53:48.463320 7094 raft_consensus.cc:2454] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbbca9ad28064f5bab856505d7dff710 in term 2.
I20260809 07:53:48.463387 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.462892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:50286)
0809 07:53:48.463217 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.463283 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":65,"threads_started":1}
I20260809 07:53:48.463642 6775 leader_election.cc:302] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: add5262818464f139ffb4c5bade4a733, bbbca9ad28064f5bab856505d7dff710; no voters:
I20260809 07:53:48.463812 7890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 07:53:48.463901 7890 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:53:48.463963 7890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 2 FOLLOWER]: Advancing to term 3
I20260809 07:53:48.464197 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.464005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:50302)
0809 07:53:48.464096 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.464146 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260809 07:53:48.464314 6780 leader_election.cc:334] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 64801c974cf64a69844c72bf20ac5983 (127.28.112.62:37291): Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
I20260809 07:53:48.465162 7890 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } }
I20260809 07:53:48.465452 7890 leader_election.cc:288] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [CANDIDATE]: Term 3 election: Requested vote from peers 64801c974cf64a69844c72bf20ac5983 (127.28.112.62:37291), add5262818464f139ffb4c5bade4a733 (127.28.112.60:41053)
I20260809 07:53:48.465772 7094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbbca9ad28064f5bab856505d7dff710" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "add5262818464f139ffb4c5bade4a733"
I20260809 07:53:48.465878 7094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 2 FOLLOWER]: Advancing to term 3
I20260809 07:53:48.465874 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.465599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:50310)
0809 07:53:48.465725 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.465813 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 07:53:48.466548 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.466350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:50326)
0809 07:53:48.466446 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.466495 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260809 07:53:48.466666 6780 leader_election.cc:334] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 64801c974cf64a69844c72bf20ac5983 (127.28.112.62:37291): Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
I20260809 07:53:48.466928 7094 raft_consensus.cc:2454] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbbca9ad28064f5bab856505d7dff710 in term 3.
I20260809 07:53:48.467192 6775 leader_election.cc:302] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: add5262818464f139ffb4c5bade4a733, bbbca9ad28064f5bab856505d7dff710; no voters: 64801c974cf64a69844c72bf20ac5983
I20260809 07:53:48.467338 7890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 07:53:48.467535 7890 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [term 3 LEADER]: Becoming Leader. State: Replica: bbbca9ad28064f5bab856505d7dff710, State: Running, Role: LEADER
I20260809 07:53:48.467784 7890 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } }
I20260809 07:53:48.469115 7894 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbbca9ad28064f5bab856505d7dff710. Latest consensus state: current_term: 3 leader_uuid: "bbbca9ad28064f5bab856505d7dff710" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } } }
I20260809 07:53:48.469221 7894 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:48.469540 7896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:53:48.472023 7896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:48.472446 7896 catalog_manager.cc:1259] Loaded cluster ID: 4b926155880248228e380cab0d664580
I20260809 07:53:48.472500 7896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:48.473054 7896 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:48.473434 7896 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: Loaded TSK: 0
I20260809 07:53:48.474109 7896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:53:48.553068 6796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:53:48.556242 6796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:53:48.559379 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.559128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:50356)
0809 07:53:48.559254 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.559312 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260809 07:53:48.559484 7094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [term 3 FOLLOWER]: Refusing update from remote peer bbbca9ad28064f5bab856505d7dff710: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260809 07:53:48.560056 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.559797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.0.0.1:50364)
0809 07:53:48.559913 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.559966 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 07:53:48.559964 7890 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260809 07:53:48.560194 6780 consensus_peers.cc:487] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 -> Peer 64801c974cf64a69844c72bf20ac5983 (127.28.112.62:37291): Couldn't send request to peer 64801c974cf64a69844c72bf20ac5983. Status: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 07:53:48.561383 7898 sys_catalog.cc:455] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbbca9ad28064f5bab856505d7dff710. Latest consensus state: current_term: 3 leader_uuid: "bbbca9ad28064f5bab856505d7dff710" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } } }
I20260809 07:53:48.561508 7898 sys_catalog.cc:458] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:53:48.562381 7897 mvcc.cc:205] Tried to move back new op lower bound from 7316529268974583808 to 7316529268606291968. Current Snapshot: MvccSnapshot[applied={T|T < 7316529268974583808}]
I20260809 07:53:48.562547 7890 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "bbbca9ad28064f5bab856505d7dff710" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } } }
I20260809 07:53:48.562657 7890 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710 [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:48.562937 7898 sys_catalog.cc:455] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "bbbca9ad28064f5bab856505d7dff710" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "64801c974cf64a69844c72bf20ac5983" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37291 } } peers { permanent_uuid: "bbbca9ad28064f5bab856505d7dff710" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 42839 } attrs { promote: false } } peers { permanent_uuid: "add5262818464f139ffb4c5bade4a733" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 41053 } attrs { promote: false } } }
I20260809 07:53:48.563056 7898 sys_catalog.cc:458] T 00000000000000000000000000000000 P add5262818464f139ffb4c5bade4a733 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:53:48.572674 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.568764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40630 (local address 127.28.112.1:43077)
0809 07:53:48.569354 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:48.569358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:48.569372 (+ 14us) server_negotiation.cc:407] Connection header received
0809 07:53:48.569423 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:48.569426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:48.569468 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:48.569543 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:48.569767 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:48.570334 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:48.571478 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:48.572100 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:48.572243 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:48.572314 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:48.572469 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 07:53:48.572545 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":71,"threads_started":1}
I20260809 07:53:48.572752 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.568169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:44345 (local address 127.0.0.1:53698)
0809 07:53:48.568305 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.568317 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:48.568434 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:48.569264 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:48.569268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:48.569286 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:48.569415 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:48.569423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:48.570536 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:48.570540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:48.571518 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:48.571525 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:48.572360 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:48.572364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:48.572475 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:48.572500 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 07:53:48.572549 (+ 49us) client_negotiation.cc:240] Negotiation successful
0809 07:53:48.572613 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":20}
I20260809 07:53:48.572642 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.568434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:43077 (local address 127.0.0.1:40630)
0809 07:53:48.568656 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.568668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:48.568752 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:48.569558 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:48.569560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:48.569576 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:48.569670 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:48.569675 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:48.570438 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:48.570441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:48.571343 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:48.571353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:48.572247 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:48.572251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:48.572337 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:48.572363 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 07:53:48.572435 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 07:53:48.572500 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":86,"threads_started":1}
I20260809 07:53:48.573077 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.568732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53698 (local address 127.28.112.3:44345)
0809 07:53:48.569011 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:48.569016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:48.569030 (+ 14us) server_negotiation.cc:407] Connection header received
0809 07:53:48.569083 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:48.569086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:48.569138 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:48.569529 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:48.569637 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:48.570417 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:48.571645 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:48.572240 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:48.572778 (+ 538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:48.572844 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:48.572878 (+ 34us) server_negotiation.cc:299] Negotiation successful
0809 07:53:48.572955 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":92,"threads_started":1}
I20260809 07:53:48.573501 7247 tablet_service.cc:1467] Processing CreateTablet for tablet 6e264899e52e444db53c5d16a7ad3642 (DEFAULT_TABLE table=test-table [id=8b9fcf2e2b544916a37ac51c986d78ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:48.573602 7670 tablet_service.cc:1467] Processing CreateTablet for tablet 6e264899e52e444db53c5d16a7ad3642 (DEFAULT_TABLE table=test-table [id=8b9fcf2e2b544916a37ac51c986d78ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:48.573794 7247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e264899e52e444db53c5d16a7ad3642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:48.573904 7670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e264899e52e444db53c5d16a7ad3642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:48.574250 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.569184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51204 (local address 127.28.112.2:39549)
0809 07:53:48.569544 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:48.569555 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:48.569571 (+ 16us) server_negotiation.cc:407] Connection header received
0809 07:53:48.569619 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:48.569622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:48.569664 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:48.569748 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:48.570158 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:48.570877 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:48.572637 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:48.573360 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:48.573460 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:48.573518 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:48.574031 (+ 513us) server_negotiation.cc:299] Negotiation successful
0809 07:53:48.574110 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":85,"threads_started":1}
I20260809 07:53:48.573940 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.569038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:39549 (local address 127.0.0.1:51204)
0809 07:53:48.569271 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.569284 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:48.569373 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:48.569856 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:48.569859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:48.569877 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:48.570032 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:48.570039 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:48.570991 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:48.570995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:48.572515 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:48.572523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:48.573580 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:48.573583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:48.573667 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:48.573690 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 07:53:48.573757 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 07:53:48.573821 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":2,"thread_start_us":98,"threads_started":1}
I20260809 07:53:48.576048 7910 tablet_bootstrap.cc:492] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc: Bootstrap starting.
I20260809 07:53:48.576200 7911 tablet_bootstrap.cc:492] T 6e264899e52e444db53c5d16a7ad3642 P 730d23ec728547609406d0fceaf73197: Bootstrap starting.
I20260809 07:53:48.576546 7460 tablet_service.cc:1467] Processing CreateTablet for tablet 6e264899e52e444db53c5d16a7ad3642 (DEFAULT_TABLE table=test-table [id=8b9fcf2e2b544916a37ac51c986d78ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:48.576807 7460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e264899e52e444db53c5d16a7ad3642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:48.577059 7910 tablet_bootstrap.cc:654] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:48.577080 7911 tablet_bootstrap.cc:654] T 6e264899e52e444db53c5d16a7ad3642 P 730d23ec728547609406d0fceaf73197: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:48.578238 7911 tablet_bootstrap.cc:492] T 6e264899e52e444db53c5d16a7ad3642 P 730d23ec728547609406d0fceaf73197: No bootstrap required, opened a new log
I20260809 07:53:48.578331 7911 ts_tablet_manager.cc:1397] T 6e264899e52e444db53c5d16a7ad3642 P 730d23ec728547609406d0fceaf73197: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:48.578747 7911 raft_consensus.cc:348] T 6e264899e52e444db53c5d16a7ad3642 P 730d23ec728547609406d0fceaf73197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } } peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } }
I20260809 07:53:48.578878 7910 tablet_bootstrap.cc:492] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc: No bootstrap required, opened a new log
I20260809 07:53:48.578956 7910 ts_tablet_manager.cc:1397] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 07:53:48.578956 7911 raft_consensus.cc:374] T 6e264899e52e444db53c5d16a7ad3642 P 730d23ec728547609406d0fceaf73197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:48.578996 7911 raft_consensus.cc:729] T 6e264899e52e444db53c5d16a7ad3642 P 730d23ec728547609406d0fceaf73197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d23ec728547609406d0fceaf73197, State: Initialized, Role: FOLLOWER
I20260809 07:53:48.579169 7911 consensus_queue.cc:260] T 6e264899e52e444db53c5d16a7ad3642 P 730d23ec728547609406d0fceaf73197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } } peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } }
I20260809 07:53:48.579360 7910 raft_consensus.cc:348] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } } peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } }
I20260809 07:53:48.579453 7910 raft_consensus.cc:374] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:48.579483 7910 raft_consensus.cc:729] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433a721039ab44cdbf224fda5eacdfdc, State: Initialized, Role: FOLLOWER
I20260809 07:53:48.579489 7911 ts_tablet_manager.cc:1428] T 6e264899e52e444db53c5d16a7ad3642 P 730d23ec728547609406d0fceaf73197: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 07:53:48.579527 7914 tablet_bootstrap.cc:492] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0: Bootstrap starting.
I20260809 07:53:48.579596 7910 consensus_queue.cc:260] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } } peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } }
I20260809 07:53:48.579906 7910 ts_tablet_manager.cc:1428] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:48.580474 7914 tablet_bootstrap.cc:654] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:48.581938 7914 tablet_bootstrap.cc:492] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0: No bootstrap required, opened a new log
I20260809 07:53:48.582054 7914 ts_tablet_manager.cc:1397] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260809 07:53:48.582543 7914 raft_consensus.cc:348] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } } peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } }
I20260809 07:53:48.582664 7914 raft_consensus.cc:374] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:48.582825 7914 raft_consensus.cc:729] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d6943ad70f4d11a5e4f6f0a1244bc0, State: Initialized, Role: FOLLOWER
I20260809 07:53:48.582845 7766 heartbeater.cc:503] Master 127.28.112.61:42839 was elected leader, sending a full tablet report...
I20260809 07:53:48.583014 7914 consensus_queue.cc:260] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } } peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } }
I20260809 07:53:48.583822 7914 ts_tablet_manager.cc:1428] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 07:53:48.586014 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.585496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.1:56563)
0809 07:53:48.585838 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.585896 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":99,"threads_started":1}
I20260809 07:53:48.586071 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.580216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.3:51291)
0809 07:53:48.581075 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.581140 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":98,"threads_started":1}
I20260809 07:53:48.586939 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.586528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.3:35003)
0809 07:53:48.586777 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.586836 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":98,"threads_started":1}
I20260809 07:53:48.587383 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.586539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.1:45309)
0809 07:53:48.587272 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.587324 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":670}
I20260809 07:53:48.587488 7568 heartbeater.cc:503] Master 127.28.112.61:42839 was elected leader, sending a full tablet report...
I20260809 07:53:48.588052 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.585063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.2:39595)
0809 07:53:48.587527 (+ 2464us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.587587 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2401,"thread_start_us":94,"threads_started":1}
I20260809 07:53:48.588872 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.588477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.2:55883)
0809 07:53:48.588730 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.588785 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":106,"threads_started":1}
I20260809 07:53:48.590144 7351 heartbeater.cc:503] Master 127.28.112.61:42839 was elected leader, sending a full tablet report...
I20260809 07:53:48.611501 7916 raft_consensus.cc:482] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:48.611622 7916 raft_consensus.cc:504] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } } peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } }
I20260809 07:53:48.612033 7916 leader_election.cc:288] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730d23ec728547609406d0fceaf73197 (127.28.112.1:43077), 94d6943ad70f4d11a5e4f6f0a1244bc0 (127.28.112.2:39549)
I20260809 07:53:48.612428 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e264899e52e444db53c5d16a7ad3642" candidate_uuid: "433a721039ab44cdbf224fda5eacdfdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d23ec728547609406d0fceaf73197" is_pre_election: true
I20260809 07:53:48.612459 7494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e264899e52e444db53c5d16a7ad3642" candidate_uuid: "433a721039ab44cdbf224fda5eacdfdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" is_pre_election: true
I20260809 07:53:48.612567 7273 raft_consensus.cc:2454] T 6e264899e52e444db53c5d16a7ad3642 P 730d23ec728547609406d0fceaf73197 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 433a721039ab44cdbf224fda5eacdfdc in term 0.
I20260809 07:53:48.612586 7494 raft_consensus.cc:2454] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 433a721039ab44cdbf224fda5eacdfdc in term 0.
I20260809 07:53:48.612895 7604 leader_election.cc:302] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 433a721039ab44cdbf224fda5eacdfdc, 730d23ec728547609406d0fceaf73197, 94d6943ad70f4d11a5e4f6f0a1244bc0; no voters:
I20260809 07:53:48.613068 7916 raft_consensus.cc:2781] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:53:48.613128 7916 raft_consensus.cc:482] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:53:48.613162 7916 raft_consensus.cc:3037] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:48.614163 7916 raft_consensus.cc:504] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } } peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } }
I20260809 07:53:48.614491 7916 leader_election.cc:288] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [CANDIDATE]: Term 1 election: Requested vote from peers 730d23ec728547609406d0fceaf73197 (127.28.112.1:43077), 94d6943ad70f4d11a5e4f6f0a1244bc0 (127.28.112.2:39549)
I20260809 07:53:48.614807 7494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e264899e52e444db53c5d16a7ad3642" candidate_uuid: "433a721039ab44cdbf224fda5eacdfdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0"
I20260809 07:53:48.614830 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e264899e52e444db53c5d16a7ad3642" candidate_uuid: "433a721039ab44cdbf224fda5eacdfdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730d23ec728547609406d0fceaf73197"
I20260809 07:53:48.614921 7494 raft_consensus.cc:3037] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:48.614953 7273 raft_consensus.cc:3037] T 6e264899e52e444db53c5d16a7ad3642 P 730d23ec728547609406d0fceaf73197 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:48.615862 7494 raft_consensus.cc:2454] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 433a721039ab44cdbf224fda5eacdfdc in term 1.
I20260809 07:53:48.615912 7273 raft_consensus.cc:2454] T 6e264899e52e444db53c5d16a7ad3642 P 730d23ec728547609406d0fceaf73197 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 433a721039ab44cdbf224fda5eacdfdc in term 1.
I20260809 07:53:48.616112 7604 leader_election.cc:302] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433a721039ab44cdbf224fda5eacdfdc, 94d6943ad70f4d11a5e4f6f0a1244bc0; no voters:
I20260809 07:53:48.616278 7916 raft_consensus.cc:2781] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:48.616484 7916 raft_consensus.cc:686] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [term 1 LEADER]: Becoming Leader. State: Replica: 433a721039ab44cdbf224fda5eacdfdc, State: Running, Role: LEADER
I20260809 07:53:48.616624 7916 consensus_queue.cc:237] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } } peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } }
I20260809 07:53:48.617163 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.616896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.3:57763)
0809 07:53:48.617054 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.617101 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260809 07:53:48.618310 6796 catalog_manager.cc:5495] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc reported cstate change: term changed from 0 to 1, leader changed from <none> to 433a721039ab44cdbf224fda5eacdfdc (127.28.112.3). New cstate: current_term: 1 leader_uuid: "433a721039ab44cdbf224fda5eacdfdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433a721039ab44cdbf224fda5eacdfdc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 } health_report { overall_health: UNKNOWN } } }
I20260809 07:53:48.634845 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.631928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.28.112.3:44345)
0809 07:53:48.632056 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:48.632061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:48.632076 (+ 15us) server_negotiation.cc:407] Connection header received
0809 07:53:48.632170 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:48.632173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:48.632220 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:48.632279 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:48.632549 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:48.633052 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:48.633848 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:48.634109 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:48.634195 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:48.634445 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:48.634516 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:48.634572 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:48.634661 (+ 89us) server_negotiation.cc:299] Negotiation successful
0809 07:53:48.634717 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 07:53:48.636786 7494 raft_consensus.cc:1261] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0 [term 1 FOLLOWER]: Refusing update from remote peer 433a721039ab44cdbf224fda5eacdfdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:48.636786 7273 raft_consensus.cc:1261] T 6e264899e52e444db53c5d16a7ad3642 P 730d23ec728547609406d0fceaf73197 [term 1 FOLLOWER]: Refusing update from remote peer 433a721039ab44cdbf224fda5eacdfdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:48.637270 7925 consensus_queue.cc:1035] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94d6943ad70f4d11a5e4f6f0a1244bc0" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:48.637423 7916 consensus_queue.cc:1035] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "730d23ec728547609406d0fceaf73197" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 43077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:48.639339 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.639118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.2:56099)
0809 07:53:48.639231 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.639283 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 07:53:48.639808 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.639539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.1:42269)
0809 07:53:48.639669 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.639735 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 07:53:48.642391 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.642147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.3:32989)
0809 07:53:48.642276 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.642326 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 07:53:48.642638 7933 mvcc.cc:205] Tried to move back new op lower bound from 7316529269292339200 to 7316529269214961664. Current Snapshot: MvccSnapshot[applied={T|T < 7316529269292339200}]
I20260809 07:53:48.643647 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.643386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.3:55413)
0809 07:53:48.643517 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.643565 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260809 07:53:48.643785 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.643506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.2:46077)
0809 07:53:48.643676 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.643725 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260809 07:53:48.644930 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.644685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37291 (local address 127.28.112.1:36541)
0809 07:53:48.644809 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.644859 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 07:53:48.646168 6794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:48.646488 6794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:48.650801 6794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbbca9ad28064f5bab856505d7dff710: Sending DeleteTablet for 3 replicas of tablet 6e264899e52e444db53c5d16a7ad3642
I20260809 07:53:48.651546 7247 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e264899e52e444db53c5d16a7ad3642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:48 UTC) from {username='oryx', principal='oryx/127.28.112.61@KRBTEST.COM'} at 127.0.0.1:40630
I20260809 07:53:48.651602 7460 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e264899e52e444db53c5d16a7ad3642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:48 UTC) from {username='oryx', principal='oryx/127.28.112.61@KRBTEST.COM'} at 127.0.0.1:51204
I20260809 07:53:48.651710 7459 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e264899e52e444db53c5d16a7ad3642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:48 UTC) from {username='oryx', principal='oryx/127.28.112.61@KRBTEST.COM'} at 127.0.0.1:51204
I20260809 07:53:48.651880 7669 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e264899e52e444db53c5d16a7ad3642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:48 UTC) from {username='oryx', principal='oryx/127.28.112.61@KRBTEST.COM'} at 127.0.0.1:53698
I20260809 07:53:48.651926 7948 tablet_replica.cc:330] T 6e264899e52e444db53c5d16a7ad3642 P 730d23ec728547609406d0fceaf73197: stopping tablet replica
I20260809 07:53:48.651913 7670 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e264899e52e444db53c5d16a7ad3642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:48 UTC) from {username='oryx', principal='oryx/127.28.112.61@KRBTEST.COM'} at 127.0.0.1:53698
I20260809 07:53:48.652045 7948 raft_consensus.cc:2229] T 6e264899e52e444db53c5d16a7ad3642 P 730d23ec728547609406d0fceaf73197 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:48.652148 7951 tablet_replica.cc:330] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc: stopping tablet replica
I20260809 07:53:48.652158 7948 raft_consensus.cc:2258] T 6e264899e52e444db53c5d16a7ad3642 P 730d23ec728547609406d0fceaf73197 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:48.652251 7951 raft_consensus.cc:2229] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:53:48.652432 7951 raft_consensus.cc:2258] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:48.652591 7950 tablet_replica.cc:330] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0: stopping tablet replica
I20260809 07:53:48.652704 7950 raft_consensus.cc:2229] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:48.652819 7950 raft_consensus.cc:2258] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:48.652930 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 7012
I20260809 07:53:48.653230 7951 ts_tablet_manager.cc:1905] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:48.653738 7950 ts_tablet_manager.cc:1905] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:48.655763 7950 ts_tablet_manager.cc:1918] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:53:48.655827 7950 log.cc:1196] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/6e264899e52e444db53c5d16a7ad3642
I20260809 07:53:48.656068 7950 ts_tablet_manager.cc:1939] T 6e264899e52e444db53c5d16a7ad3642 P 94d6943ad70f4d11a5e4f6f0a1244bc0: Deleting consensus metadata
I20260809 07:53:48.656335 7951 ts_tablet_manager.cc:1918] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:53:48.656436 7951 log.cc:1196] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/6e264899e52e444db53c5d16a7ad3642
I20260809 07:53:48.656646 6780 catalog_manager.cc:4841] TS 94d6943ad70f4d11a5e4f6f0a1244bc0 (127.28.112.2:39549): tablet 6e264899e52e444db53c5d16a7ad3642 (table test-table [id=8b9fcf2e2b544916a37ac51c986d78ef]) successfully deleted
I20260809 07:53:48.656721 7951 ts_tablet_manager.cc:1939] T 6e264899e52e444db53c5d16a7ad3642 P 433a721039ab44cdbf224fda5eacdfdc: Deleting consensus metadata
W20260809 07:53:48.657096 6780 catalog_manager.cc:4808] TS 94d6943ad70f4d11a5e4f6f0a1244bc0 (127.28.112.2:39549): delete failed for tablet 6e264899e52e444db53c5d16a7ad3642 because the tablet was not found. No further retry: Not found: Tablet not found: 6e264899e52e444db53c5d16a7ad3642
I20260809 07:53:48.657394 6778 catalog_manager.cc:4841] TS 433a721039ab44cdbf224fda5eacdfdc (127.28.112.3:44345): tablet 6e264899e52e444db53c5d16a7ad3642 (table test-table [id=8b9fcf2e2b544916a37ac51c986d78ef]) successfully deleted
W20260809 07:53:48.657665 6778 catalog_manager.cc:4808] TS 433a721039ab44cdbf224fda5eacdfdc (127.28.112.3:44345): delete failed for tablet 6e264899e52e444db53c5d16a7ad3642 because the tablet was not found. No further retry: Not found: Tablet not found: 6e264899e52e444db53c5d16a7ad3642
I20260809 07:53:48.663580 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 7364
I20260809 07:53:48.663707 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:48.663474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:43077 (local address 127.0.0.1:49130)
0809 07:53:48.663585 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:48.663636 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:43077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 07:53:48.671195 6775 catalog_manager.cc:4584] TS 730d23ec728547609406d0fceaf73197 (127.28.112.1:43077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6e264899e52e444db53c5d16a7ad3642: Network error: Client connection negotiation failed: client connection to 127.28.112.1:43077: connect: Connection refused (error 111)
I20260809 07:53:48.676014 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 7573
I20260809 07:53:48.690012 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 6592
I20260809 07:53:48.697342 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 6746
2026-08-09T07:53:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7369 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:53:48 dist-test-slave-vpvm krb5kdc[7959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:53:48 dist-test-slave-vpvm krb5kdc[7959](info): set up 2 sockets
Aug 09 07:53:48 dist-test-slave-vpvm krb5kdc[7959](info): commencing operation
krb5kdc: starting...
W20260809 07:53:49.829114 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:53:49 dist-test-slave-vpvm krb5kdc[7959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:53:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:53:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.62.keytab.
Entry for principal oryx/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.62.keytab.
I20260809 07:53:49.946323 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:45347
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.62.keytab
--principal=oryx/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:39581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:45347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:50.054972 7973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:50.055205 7973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:50.055258 7973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:50.059115 7973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:50.059165 7973 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 07:53:50.059186 7973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:50.059209 7973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:50.059231 7973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:50.059259 7973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:53:50.064308 7973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:39581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:45347
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.62.keytab
--principal=oryx/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:45347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:50.065263 7973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:50.066252 7973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:50.074599 7985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:50.074648 7989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:50.074662 7973 server_base.cc:1034] running on GCE node
W20260809 07:53:50.074599 7984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:50.075317 7973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:50.076375 7973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:50.077536 7973 hybrid_clock.cc:648] HybridClock initialized: now 1786262030077519 us; error 28 us; skew 500 ppm
Aug 09 07:53:50 dist-test-slave-vpvm krb5kdc[7959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262030, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:50.080113 7973 init.cc:377] Logged in from keytab as oryx/127.28.112.62@KRBTEST.COM (short username oryx)
I20260809 07:53:50.080972 7973 webserver.cc:460] Webserver started at http://127.28.112.62:35713/ using document root <none> and password file <none>
I20260809 07:53:50.081493 7973 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:50.081560 7973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:50.081771 7973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:50.083786 7973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "3195bf3fdf00439f97686ff97571eb4b"
format_stamp: "Formatted at 2026-08-09 07:53:50 on dist-test-slave-vpvm"
server_key: "b1a9b7fd501a7be509d1eaaf621fecb8"
server_key_iv: "9d29a9604a0871e3e0baf8c8b778b552"
server_key_version: "encryptionkey@0"
I20260809 07:53:50.084298 7973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "3195bf3fdf00439f97686ff97571eb4b"
format_stamp: "Formatted at 2026-08-09 07:53:50 on dist-test-slave-vpvm"
server_key: "b1a9b7fd501a7be509d1eaaf621fecb8"
server_key_iv: "9d29a9604a0871e3e0baf8c8b778b552"
server_key_version: "encryptionkey@0"
I20260809 07:53:50.087831 7973 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 07:53:50.090077 7999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:50.091189 7973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:50.091306 7973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "3195bf3fdf00439f97686ff97571eb4b"
format_stamp: "Formatted at 2026-08-09 07:53:50 on dist-test-slave-vpvm"
server_key: "b1a9b7fd501a7be509d1eaaf621fecb8"
server_key_iv: "9d29a9604a0871e3e0baf8c8b778b552"
server_key_version: "encryptionkey@0"
I20260809 07:53:50.091398 7973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:50.106438 7973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:50.109023 7973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:50.109193 7973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:50.116780 7973 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:45347
I20260809 07:53:50.116787 8089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:45347 every 8 connection(s)
I20260809 07:53:50.117664 7973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:53:50.120360 8090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:50.122002 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 7973
I20260809 07:53:50.122121 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:53:50.122416 29120 external_mini_cluster.cc:1428] Setting key 9b839dd77a3051cf23fbc0854835c692
I20260809 07:53:50.126621 8090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b: Bootstrap starting.
I20260809 07:53:50.129092 8090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:50.129985 8090 log.cc:824] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b: Log is configured to *not* fsync() on all Append() calls
Aug 09 07:53:50 dist-test-slave-vpvm krb5kdc[7959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM
I20260809 07:53:50.132160 8090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b: No bootstrap required, opened a new log
I20260809 07:53:50.136613 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.123722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.28.112.62:45347)
0809 07:53:50.124097 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:50.124112 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:50.124148 (+ 36us) server_negotiation.cc:407] Connection header received
0809 07:53:50.124782 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:50.124800 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:50.125106 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:50.125371 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:50.126291 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.127295 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.128431 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.128871 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.128978 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.131813 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:50.131818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:50.131831 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:50.131856 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:50.133651 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:50.134088 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:50.134095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:50.134102 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:50.134238 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:50.134508 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:50.134511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:50.134513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:50.134991 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:50.135205 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:50.135361 (+ 156us) server_negotiation.cc:299] Negotiation successful
0809 07:53:50.135677 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":105,"threads_started":1}
I20260809 07:53:50.136415 8090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195bf3fdf00439f97686ff97571eb4b" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45347 } }
I20260809 07:53:50.136715 8090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:50.136772 8090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3195bf3fdf00439f97686ff97571eb4b, State: Initialized, Role: FOLLOWER
I20260809 07:53:50.137248 8090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195bf3fdf00439f97686ff97571eb4b" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45347 } }
I20260809 07:53:50.137405 8090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:53:50.137470 8090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:53:50.137566 8090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:50.138429 8090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195bf3fdf00439f97686ff97571eb4b" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45347 } }
I20260809 07:53:50.138751 8090 leader_election.cc:302] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3195bf3fdf00439f97686ff97571eb4b; no voters:
I20260809 07:53:50.139050 8090 leader_election.cc:288] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:53:50.139145 8098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:50.139358 8098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b [term 1 LEADER]: Becoming Leader. State: Replica: 3195bf3fdf00439f97686ff97571eb4b, State: Running, Role: LEADER
I20260809 07:53:50.139648 8098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195bf3fdf00439f97686ff97571eb4b" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45347 } }
I20260809 07:53:50.140139 8090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:50.141149 8100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3195bf3fdf00439f97686ff97571eb4b. Latest consensus state: current_term: 1 leader_uuid: "3195bf3fdf00439f97686ff97571eb4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195bf3fdf00439f97686ff97571eb4b" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45347 } } }
I20260809 07:53:50.141273 8100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:50.141491 8099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3195bf3fdf00439f97686ff97571eb4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3195bf3fdf00439f97686ff97571eb4b" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45347 } } }
I20260809 07:53:50.141583 8099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b [sys.catalog]: This master's current role is: LEADER
W20260809 07:53:50.144590 8121 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:53:50.144660 8121 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:53:50.144735 8118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:53:50.145390 8118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:50.150053 8118 catalog_manager.cc:1347] Generated new cluster ID: cbcbcb69a2bc479dba8ba9d6f9591eab
I20260809 07:53:50.150113 8118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:50.158633 8118 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:53:50.159578 8118 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:50.177842 8118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b: Generated new TSK 0
I20260809 07:53:50.178725 8118 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.1.keytab.
Entry for principal oryx/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.1.keytab.
I20260809 07:53:50.213336 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:45347
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.1.keytab
--principal=oryx/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:39581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:50.326092 8128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:50.326347 8128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:50.326390 8128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:50.330408 8128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:50.330464 8128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:50.330502 8128 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 07:53:50.330540 8128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:53:50.336042 8128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:39581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.1.keytab
--principal=oryx/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.112.62:45347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:50.337080 8128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:50.338117 8128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:50.347723 8133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:50.347741 8138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:50.347779 8128 server_base.cc:1034] running on GCE node
W20260809 07:53:50.347715 8136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:50.348318 8128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:50.348943 8128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:50.350097 8128 hybrid_clock.cc:648] HybridClock initialized: now 1786262030350071 us; error 34 us; skew 500 ppm
Aug 09 07:53:50 dist-test-slave-vpvm krb5kdc[7959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262030, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:50.352818 8128 init.cc:377] Logged in from keytab as oryx/127.28.112.1@KRBTEST.COM (short username oryx)
I20260809 07:53:50.353713 8128 webserver.cc:460] Webserver started at http://127.28.112.1:38393/ using document root <none> and password file <none>
I20260809 07:53:50.354292 8128 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:50.354365 8128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:50.354588 8128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:50.356570 8128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "a3f286b3371745e88d4efbcec859a619"
format_stamp: "Formatted at 2026-08-09 07:53:50 on dist-test-slave-vpvm"
server_key: "695264179c6d3ccb162c01e8b763247e"
server_key_iv: "8487a1c254ddc3c2be9309e818684904"
server_key_version: "encryptionkey@0"
I20260809 07:53:50.357129 8128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "a3f286b3371745e88d4efbcec859a619"
format_stamp: "Formatted at 2026-08-09 07:53:50 on dist-test-slave-vpvm"
server_key: "695264179c6d3ccb162c01e8b763247e"
server_key_iv: "8487a1c254ddc3c2be9309e818684904"
server_key_version: "encryptionkey@0"
I20260809 07:53:50.360991 8128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 07:53:50.363410 8146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:50.364560 8128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:50.364671 8128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "a3f286b3371745e88d4efbcec859a619"
format_stamp: "Formatted at 2026-08-09 07:53:50 on dist-test-slave-vpvm"
server_key: "695264179c6d3ccb162c01e8b763247e"
server_key_iv: "8487a1c254ddc3c2be9309e818684904"
server_key_version: "encryptionkey@0"
I20260809 07:53:50.364773 8128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:50.385634 8128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:50.388840 8128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:50.389031 8128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:50.390625 8128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:50.390676 8128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:50.390717 8128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:50.390746 8128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 07:53:50 dist-test-slave-vpvm krb5kdc[7959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262030, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM
I20260809 07:53:50.401047 8128 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:42231
I20260809 07:53:50.401087 8321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:42231 every 8 connection(s)
I20260809 07:53:50.402060 8128 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:53:50.403522 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.391305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:43539 (local address 127.28.112.62:45347)
0809 07:53:50.391465 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:50.391470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:50.392311 (+ 841us) server_negotiation.cc:407] Connection header received
0809 07:53:50.393279 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:50.393283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:50.393346 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:50.393405 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:50.394961 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.395622 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.396812 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.397078 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.397189 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.399904 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:50.399908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:50.399910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:50.399937 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:50.401426 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:50.401898 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:50.401902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:50.401904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:50.402038 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:50.402389 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:50.402393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:50.402394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:50.402606 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:50.402704 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:50.403290 (+ 586us) server_negotiation.cc:299] Negotiation successful
0809 07:53:50.403400 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 07:53:50.404247 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.391698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:45347 (local address 127.28.112.1:43539)
0809 07:53:50.392146 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:50.392197 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:50.393044 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:50.393812 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:50.393819 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:50.394233 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:50.394818 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:50.394835 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.395742 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.395745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:50.396690 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:50.396701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.397348 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.397351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:50.397497 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.398068 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:50.398091 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:50.399745 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:50.401533 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:50.401538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:50.401548 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:50.401797 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:50.402137 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:50.402141 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:50.402143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:50.402292 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:50.402759 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:50.402765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:50.403037 (+ 272us) client_negotiation.cc:769] Sending connection context
0809 07:53:50.403227 (+ 190us) client_negotiation.cc:240] Negotiation successful
0809 07:53:50.403466 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":101,"threads_started":1}
I20260809 07:53:50.405514 8322 heartbeater.cc:347] Connected to a master server at 127.28.112.62:45347
I20260809 07:53:50.405658 8322 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:50.406140 8322 heartbeater.cc:511] Master 127.28.112.62:45347 requested a full tablet report, sending...
I20260809 07:53:50.407598 8025 ts_manager.cc:194] Registered new tserver with Master: a3f286b3371745e88d4efbcec859a619 (127.28.112.1:42231)
I20260809 07:53:50.408919 8025 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:43539
I20260809 07:53:50.409662 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 8128
I20260809 07:53:50.409777 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:53:50.410084 29120 external_mini_cluster.cc:1428] Setting key 43784e3db64716e13c062bc29d490e54
I20260809 07:53:50.415567 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.407467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:45347 (local address 127.28.112.1:59195)
0809 07:53:50.407746 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:50.407760 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:50.407857 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:50.408061 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:50.408063 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:50.408254 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:50.408368 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:50.408375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.409100 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.409103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:50.411260 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:50.411270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.411764 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.411767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:50.411833 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.412252 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:50.412262 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:50.412581 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:50.414147 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:50.414150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:50.414152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:50.414386 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:50.414738 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:50.414741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:50.414743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:50.414829 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:50.415187 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:50.415189 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:50.415258 (+ 69us) client_negotiation.cc:769] Sending connection context
0809 07:53:50.415331 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 07:53:50.415427 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":105,"threads_started":1}
I20260809 07:53:50.415655 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.407601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:59195 (local address 127.28.112.62:45347)
0809 07:53:50.407724 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:50.407728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:50.407815 (+ 87us) server_negotiation.cc:407] Connection header received
0809 07:53:50.407932 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:50.407934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:50.407968 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:50.408027 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:50.408476 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.408977 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.411365 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.411665 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.411771 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.412716 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:50.412720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:50.412723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:50.412748 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:50.414049 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:50.414560 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:50.414564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:50.414565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:50.414658 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:50.414908 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:50.414911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:50.414912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:50.415110 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:50.415205 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:50.415395 (+ 190us) server_negotiation.cc:299] Negotiation successful
0809 07:53:50.415503 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":56}
WARNING: no policy specified for oryx/127.28.112.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.112.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.2.keytab.
Entry for principal oryx/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.2.keytab.
I20260809 07:53:50.436272 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:45347
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.2.keytab
--principal=oryx/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:39581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:50.546319 8334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:50.546595 8334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:50.546646 8334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:50.550525 8334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:50.550587 8334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:50.550626 8334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 07:53:50.550660 8334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:53:50.555819 8334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:39581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.2.keytab
--principal=oryx/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.112.62:45347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:50.556845 8334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:50.557845 8334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:50.567399 8346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:50.567440 8341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:50.567433 8340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:50.567667 8334 server_base.cc:1034] running on GCE node
I20260809 07:53:50.568172 8334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:50.568874 8334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:50.570052 8334 hybrid_clock.cc:648] HybridClock initialized: now 1786262030570027 us; error 36 us; skew 500 ppm
Aug 09 07:53:50 dist-test-slave-vpvm krb5kdc[7959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262030, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:50.572938 8334 init.cc:377] Logged in from keytab as oryx/127.28.112.2@KRBTEST.COM (short username oryx)
I20260809 07:53:50.574062 8334 webserver.cc:460] Webserver started at http://127.28.112.2:44867/ using document root <none> and password file <none>
I20260809 07:53:50.574648 8334 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:50.574719 8334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:50.574925 8334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:50.576952 8334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "1813e38e916647beb49a55a5bc392f01"
format_stamp: "Formatted at 2026-08-09 07:53:50 on dist-test-slave-vpvm"
server_key: "bfd162f177518a17ee8d501922e3b673"
server_key_iv: "d98542de6371b5dad86601496a4c6341"
server_key_version: "encryptionkey@0"
I20260809 07:53:50.577592 8334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "1813e38e916647beb49a55a5bc392f01"
format_stamp: "Formatted at 2026-08-09 07:53:50 on dist-test-slave-vpvm"
server_key: "bfd162f177518a17ee8d501922e3b673"
server_key_iv: "d98542de6371b5dad86601496a4c6341"
server_key_version: "encryptionkey@0"
I20260809 07:53:50.581207 8334 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:53:50.583649 8359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:50.584843 8334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:50.584982 8334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "1813e38e916647beb49a55a5bc392f01"
format_stamp: "Formatted at 2026-08-09 07:53:50 on dist-test-slave-vpvm"
server_key: "bfd162f177518a17ee8d501922e3b673"
server_key_iv: "d98542de6371b5dad86601496a4c6341"
server_key_version: "encryptionkey@0"
I20260809 07:53:50.585081 8334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:50.604800 8334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:50.608161 8334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:50.608354 8334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:50.609846 8334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:50.609910 8334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:50.609992 8334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:50.610023 8334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 07:53:50 dist-test-slave-vpvm krb5kdc[7959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262030, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM
I20260809 07:53:50.620898 8334 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:45051
I20260809 07:53:50.620954 8532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:45051 every 8 connection(s)
I20260809 07:53:50.622004 8334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:53:50.623126 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.610578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:52643 (local address 127.28.112.62:45347)
0809 07:53:50.610739 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:50.610745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:50.611520 (+ 775us) server_negotiation.cc:407] Connection header received
0809 07:53:50.612561 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:50.612564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:50.612622 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:50.612700 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:50.614204 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.614790 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.616070 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.616297 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.616385 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.619312 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:50.619317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:50.619320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:50.619351 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:50.620844 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:50.621455 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:50.621458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:50.621459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:50.621535 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:50.621941 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:50.621944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:50.621969 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:50.622191 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:50.622273 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:50.622836 (+ 563us) server_negotiation.cc:299] Negotiation successful
0809 07:53:50.622959 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 07:53:50.623873 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.610925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:45347 (local address 127.28.112.2:52643)
0809 07:53:50.611372 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:50.611422 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:50.612358 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:50.612983 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:50.612993 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:50.613445 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:50.614058 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:50.614076 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.614943 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.614947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:50.615952 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:50.615963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.616436 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.616440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:50.616599 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.617125 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:50.617143 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:50.619130 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:50.620987 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:50.620993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:50.621007 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:50.621335 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:50.621627 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:50.621631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:50.621633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:50.621804 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:50.622297 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:50.622304 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:50.622608 (+ 304us) client_negotiation.cc:769] Sending connection context
0809 07:53:50.622782 (+ 174us) client_negotiation.cc:240] Negotiation successful
0809 07:53:50.623098 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":97,"threads_started":1}
I20260809 07:53:50.625113 8536 heartbeater.cc:347] Connected to a master server at 127.28.112.62:45347
I20260809 07:53:50.625267 8536 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:50.625746 8536 heartbeater.cc:511] Master 127.28.112.62:45347 requested a full tablet report, sending...
I20260809 07:53:50.626879 8024 ts_manager.cc:194] Registered new tserver with Master: 1813e38e916647beb49a55a5bc392f01 (127.28.112.2:45051)
I20260809 07:53:50.627671 8024 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:52643
I20260809 07:53:50.632215 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 8334
I20260809 07:53:50.632342 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:53:50.632674 29120 external_mini_cluster.cc:1428] Setting key 95fb48db5d7ba03dc4a77a3308c99c59
I20260809 07:53:50.635053 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.627995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:35367 (local address 127.28.112.62:45347)
0809 07:53:50.628098 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:50.628102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:50.628116 (+ 14us) server_negotiation.cc:407] Connection header received
0809 07:53:50.628151 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:50.628154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:50.628192 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:50.628267 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:50.628929 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.629669 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.630881 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.631138 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.631197 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.632167 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:50.632171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:50.632174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:50.632198 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:50.633566 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:50.633942 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:50.633945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:50.633967 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:50.634052 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:50.634323 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:50.634326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:50.634327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:50.634521 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:50.634599 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:50.634848 (+ 249us) server_negotiation.cc:299] Negotiation successful
0809 07:53:50.634944 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 07:53:50.635114 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.627081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:45347 (local address 127.28.112.2:35367)
0809 07:53:50.627333 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:50.627348 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:50.627446 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:50.628380 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:50.628383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:50.628634 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:50.628755 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:50.628762 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.629804 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.629808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:50.630732 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:50.630743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.631317 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.631320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:50.631387 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.631755 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:50.631765 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:50.632042 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:50.633661 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:50.633663 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:50.633665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:50.633849 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:50.634135 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:50.634138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:50.634139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:50.634236 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:50.634611 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:50.634614 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:50.634708 (+ 94us) client_negotiation.cc:769] Sending connection context
0809 07:53:50.634810 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 07:53:50.634933 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for oryx/127.28.112.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.112.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.3.keytab.
Entry for principal oryx/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.3.keytab.
I20260809 07:53:50.658635 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:45347
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.3.keytab
--principal=oryx/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:39581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:50.767825 8550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:50.768090 8550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:50.768159 8550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:50.772087 8550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:50.772149 8550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:50.772213 8550 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 07:53:50.772271 8550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:53:50.777244 8550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:39581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.3.keytab
--principal=oryx/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.112.62:45347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:50.778223 8550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:50.779197 8550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:50.788278 8562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:50.788290 8564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:50.788264 8559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:50.788564 8550 server_base.cc:1034] running on GCE node
I20260809 07:53:50.788985 8550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:50.789561 8550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:50.790720 8550 hybrid_clock.cc:648] HybridClock initialized: now 1786262030790692 us; error 41 us; skew 500 ppm
Aug 09 07:53:50 dist-test-slave-vpvm krb5kdc[7959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262030, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:50.793370 8550 init.cc:377] Logged in from keytab as oryx/127.28.112.3@KRBTEST.COM (short username oryx)
I20260809 07:53:50.794237 8550 webserver.cc:460] Webserver started at http://127.28.112.3:39031/ using document root <none> and password file <none>
I20260809 07:53:50.794768 8550 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:50.794832 8550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:50.795033 8550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:50.796974 8550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276"
format_stamp: "Formatted at 2026-08-09 07:53:50 on dist-test-slave-vpvm"
server_key: "e3445cca6439ff5d46b08e0019cd96dc"
server_key_iv: "6fc9366a2647e564e49557d9a416ca15"
server_key_version: "encryptionkey@0"
I20260809 07:53:50.797492 8550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276"
format_stamp: "Formatted at 2026-08-09 07:53:50 on dist-test-slave-vpvm"
server_key: "e3445cca6439ff5d46b08e0019cd96dc"
server_key_iv: "6fc9366a2647e564e49557d9a416ca15"
server_key_version: "encryptionkey@0"
I20260809 07:53:50.800940 8550 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:53:50.803217 8574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:50.804311 8550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:50.804430 8550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276"
format_stamp: "Formatted at 2026-08-09 07:53:50 on dist-test-slave-vpvm"
server_key: "e3445cca6439ff5d46b08e0019cd96dc"
server_key_iv: "6fc9366a2647e564e49557d9a416ca15"
server_key_version: "encryptionkey@0"
I20260809 07:53:50.804517 8550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:50.821827 8550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:50.824618 8550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:50.824798 8550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:50.826359 8550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:50.826412 8550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:50.826467 8550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:50.826494 8550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 07:53:50 dist-test-slave-vpvm krb5kdc[7959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262030, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM
I20260809 07:53:50.836913 8550 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:39831
I20260809 07:53:50.837359 8745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:39831 every 8 connection(s)
I20260809 07:53:50.837901 8550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:53:50.840524 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.827128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:55817 (local address 127.28.112.62:45347)
0809 07:53:50.827241 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:50.827245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:50.828253 (+ 1008us) server_negotiation.cc:407] Connection header received
0809 07:53:50.829245 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:50.829249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:50.829316 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:50.829408 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:50.831032 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.831804 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.833445 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.833816 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.833929 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.836951 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:50.836954 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:50.836957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:50.836991 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:50.838694 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:50.839204 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:50.839207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:50.839209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:50.839299 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:50.839559 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:50.839561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:50.839563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:50.839740 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:50.839826 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:50.840218 (+ 392us) server_negotiation.cc:299] Negotiation successful
0809 07:53:50.840326 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 07:53:50.841320 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.827616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:45347 (local address 127.28.112.3:55817)
0809 07:53:50.828109 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:50.828151 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:50.828996 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:50.829731 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:50.829740 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:50.830222 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:50.830884 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:50.830906 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.831938 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.831942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:50.833311 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:50.833324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.834000 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.834005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:50.834201 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.834824 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:50.834845 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:50.836737 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:50.838821 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:50.838826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:50.838839 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:50.839110 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:50.839378 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:50.839380 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:50.839381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:50.839495 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:50.839820 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:50.839825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:50.840033 (+ 208us) client_negotiation.cc:769] Sending connection context
0809 07:53:50.840174 (+ 141us) client_negotiation.cc:240] Negotiation successful
0809 07:53:50.840504 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":134,"threads_started":1}
I20260809 07:53:50.842677 8746 heartbeater.cc:347] Connected to a master server at 127.28.112.62:45347
I20260809 07:53:50.842819 8746 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:50.843284 8746 heartbeater.cc:511] Master 127.28.112.62:45347 requested a full tablet report, sending...
I20260809 07:53:50.844113 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 8550
I20260809 07:53:50.844233 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:53:50.844364 8025 ts_manager.cc:194] Registered new tserver with Master: 5ef4fcc7c19e4c7ca825f7930d1ae276 (127.28.112.3:39831)
I20260809 07:53:50.844509 29120 external_mini_cluster.cc:1428] Setting key c96e76e04e13d5776c9aa42a33e7bcf6
I20260809 07:53:50.845104 8025 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:55817
I20260809 07:53:50.846519 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 07:53:50.851992 8752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.844749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:45347 (local address 127.28.112.3:45709)
0809 07:53:50.845016 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:50.845039 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:50.845120 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:50.845579 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:50.845582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:50.845793 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:50.845923 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:50.845930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.846967 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.846971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:50.847723 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:50.847735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.848131 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.848134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:50.848218 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.848628 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:50.848642 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:50.848966 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:50.850619 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:50.850622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:50.850623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:50.850823 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:50.851103 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:50.851105 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:50.851106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:50.851177 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:50.851546 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:50.851548 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:50.851627 (+ 79us) client_negotiation.cc:769] Sending connection context
0809 07:53:50.851714 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 07:53:50.851815 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":102,"threads_started":1}
I20260809 07:53:50.852080 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.844807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:45709 (local address 127.28.112.62:45347)
0809 07:53:50.845396 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:50.845399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:50.845419 (+ 20us) server_negotiation.cc:407] Connection header received
0809 07:53:50.845445 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:50.845447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:50.845476 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:50.845541 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:50.846100 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.846828 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.847833 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.848037 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.848110 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.849105 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:50.849108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:50.849111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:50.849136 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:50.850486 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:50.850924 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:50.850928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:50.850930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:50.851016 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:50.851247 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:50.851250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:50.851252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:50.851435 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:50.851535 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:50.851795 (+ 260us) server_negotiation.cc:299] Negotiation successful
0809 07:53:50.851907 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524}
Aug 09 07:53:50 dist-test-slave-vpvm krb5kdc[7959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262030, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 07:53:50 dist-test-slave-vpvm krb5kdc[7959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:53:50 dist-test-slave-vpvm krb5kdc[7959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 07:53:50.870987 8096 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:50.865294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46060 (local address 127.28.112.62:45347)
0809 07:53:50.865414 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:50.865417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:50.865427 (+ 10us) server_negotiation.cc:407] Connection header received
0809 07:53:50.865456 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:50.865458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:50.865491 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:50.865550 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:50.866248 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.866734 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.867528 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.867730 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.867813 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.870852 (+ 3039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46060: BlockingRecv error: recv got EOF from 127.0.0.1:46060 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 09 07:53:50 dist-test-slave-vpvm krb5kdc[7959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262030, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM
I20260809 07:53:50.879217 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.872307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.28.112.62:45347)
0809 07:53:50.872412 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:50.872416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:50.872584 (+ 168us) server_negotiation.cc:407] Connection header received
0809 07:53:50.872689 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:50.872692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:50.872729 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:50.872789 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:50.873349 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.873816 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.874583 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.874786 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.874867 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.876868 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:50.876872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:50.876874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:50.876895 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:50.877975 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:50.878381 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:50.878384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:50.878385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:50.878455 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:50.878643 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:50.878644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:50.878646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:50.878789 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:50.878879 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:50.879023 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 07:53:50.879114 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 07:53:50.881793 8025 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:53:50.884850 8025 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:53:50.898633 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.892839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50338 (local address 127.28.112.1:42231)
0809 07:53:50.893997 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:50.894014 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:50.894038 (+ 24us) server_negotiation.cc:407] Connection header received
0809 07:53:50.894118 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:50.894123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:50.894382 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:50.894518 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:50.894816 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.895583 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.897170 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.897992 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.898107 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.898274 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:50.898396 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 07:53:50.898484 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1004,"thread_start_us":67,"threads_started":1}
I20260809 07:53:50.898639 8776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.892669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:42231 (local address 127.0.0.1:50338)
0809 07:53:50.893693 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:50.893706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:50.893772 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:50.894519 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:50.894525 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:50.894546 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:50.894688 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:50.894696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.895721 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.895724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:50.897044 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:50.897055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.898106 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.898109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:50.898216 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.898254 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 07:53:50.898364 (+ 110us) client_negotiation.cc:240] Negotiation successful
0809 07:53:50.898484 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":26,"thread_start_us":80,"threads_started":1}
I20260809 07:53:50.899037 8779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.893254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:39831 (local address 127.0.0.1:47000)
0809 07:53:50.894131 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:50.894142 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:50.894232 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:50.894762 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:50.894765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:50.894779 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:50.894871 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:50.894878 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.896345 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.896348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:50.897418 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:50.897428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.898681 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.898684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:50.898772 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.898796 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 07:53:50.898862 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 07:53:50.898921 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":22,"thread_start_us":92,"threads_started":1}
I20260809 07:53:50.899147 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.893407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.28.112.3:39831)
0809 07:53:50.894002 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:50.894014 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:50.894222 (+ 208us) server_negotiation.cc:407] Connection header received
0809 07:53:50.894323 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:50.894327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:50.894474 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:50.894624 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:50.895063 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.896222 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.897555 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.898355 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.898464 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.898617 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:50.898905 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 07:53:50.899000 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":105,"threads_started":1}
I20260809 07:53:50.899683 8781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.893430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59868 (local address 127.28.112.2:45051)
0809 07:53:50.894082 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:50.894095 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:50.894119 (+ 24us) server_negotiation.cc:407] Connection header received
0809 07:53:50.894193 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:50.894198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:50.894384 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:50.894673 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:50.894829 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.895967 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.897289 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.898114 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.899004 (+ 890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.899417 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:50.899469 (+ 52us) server_negotiation.cc:299] Negotiation successful
0809 07:53:50.899550 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":125,"threads_started":1}
I20260809 07:53:50.901366 8223 tablet_service.cc:1467] Processing CreateTablet for tablet cda9e05b4d014830bd96c002d5ea6108 (DEFAULT_TABLE table=test-table [id=e052ef6be1a44db1959ad2c44c38ccba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:50.901420 8641 tablet_service.cc:1467] Processing CreateTablet for tablet cda9e05b4d014830bd96c002d5ea6108 (DEFAULT_TABLE table=test-table [id=e052ef6be1a44db1959ad2c44c38ccba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:50.898644 8778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:50.892814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:45051 (local address 127.0.0.1:59868)
0809 07:53:50.893496 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:50.893553 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:50.893676 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:50.894585 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:50.894589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:50.894608 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:50.894746 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:50.894753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.896089 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.896093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:50.897175 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:50.897182 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:50.898257 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:50.898261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:50.898349 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:50.898374 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 07:53:50.898421 (+ 47us) client_negotiation.cc:240] Negotiation successful
0809 07:53:50.898492 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":37,"thread_start_us":104,"threads_started":1}
I20260809 07:53:50.902354 8223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda9e05b4d014830bd96c002d5ea6108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:50.902377 8641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda9e05b4d014830bd96c002d5ea6108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:50.904726 8440 tablet_service.cc:1467] Processing CreateTablet for tablet cda9e05b4d014830bd96c002d5ea6108 (DEFAULT_TABLE table=test-table [id=e052ef6be1a44db1959ad2c44c38ccba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:50.905731 8440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda9e05b4d014830bd96c002d5ea6108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:50.909237 8782 tablet_bootstrap.cc:492] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276: Bootstrap starting.
I20260809 07:53:50.910444 8783 tablet_bootstrap.cc:492] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619: Bootstrap starting.
I20260809 07:53:50.911593 8782 tablet_bootstrap.cc:654] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:50.912400 8785 tablet_bootstrap.cc:492] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01: Bootstrap starting.
I20260809 07:53:50.912461 8782 log.cc:824] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:50.912926 8783 tablet_bootstrap.cc:654] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:50.913620 8783 log.cc:824] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:50.914340 8785 tablet_bootstrap.cc:654] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:50.915112 8785 log.cc:824] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:50.915520 8782 tablet_bootstrap.cc:492] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276: No bootstrap required, opened a new log
I20260809 07:53:50.915712 8782 ts_tablet_manager.cc:1397] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260809 07:53:50.915750 8783 tablet_bootstrap.cc:492] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619: No bootstrap required, opened a new log
I20260809 07:53:50.915920 8783 ts_tablet_manager.cc:1397] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 07:53:50.917290 8785 tablet_bootstrap.cc:492] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01: No bootstrap required, opened a new log
I20260809 07:53:50.917467 8785 ts_tablet_manager.cc:1397] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 07:53:50.918716 8782 raft_consensus.cc:348] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:50.918928 8782 raft_consensus.cc:374] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:50.918991 8782 raft_consensus.cc:729] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ef4fcc7c19e4c7ca825f7930d1ae276, State: Initialized, Role: FOLLOWER
I20260809 07:53:50.919379 8782 consensus_queue.cc:260] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:50.919435 8783 raft_consensus.cc:348] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:50.919691 8783 raft_consensus.cc:374] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:50.919751 8783 raft_consensus.cc:729] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f286b3371745e88d4efbcec859a619, State: Initialized, Role: FOLLOWER
I20260809 07:53:50.920090 8746 heartbeater.cc:503] Master 127.28.112.62:45347 was elected leader, sending a full tablet report...
I20260809 07:53:50.920163 8783 consensus_queue.cc:260] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:50.920385 8782 ts_tablet_manager.cc:1428] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260809 07:53:50.921001 8783 ts_tablet_manager.cc:1428] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 07:53:50.921075 8785 raft_consensus.cc:348] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:50.921289 8785 raft_consensus.cc:374] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:50.921339 8785 raft_consensus.cc:729] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1813e38e916647beb49a55a5bc392f01, State: Initialized, Role: FOLLOWER
I20260809 07:53:50.921357 8322 heartbeater.cc:503] Master 127.28.112.62:45347 was elected leader, sending a full tablet report...
I20260809 07:53:50.921736 8785 consensus_queue.cc:260] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:50.922322 8536 heartbeater.cc:503] Master 127.28.112.62:45347 was elected leader, sending a full tablet report...
I20260809 07:53:50.922628 8785 ts_tablet_manager.cc:1428] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260809 07:53:51.089037 8747 tablet.cc:2369] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 07:53:51.123121 8537 tablet.cc:2369] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 07:53:51.153340 8323 tablet.cc:2369] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:53:51.198928 8789 raft_consensus.cc:482] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:51.199154 8789 raft_consensus.cc:504] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.200312 8789 leader_election.cc:288] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3f286b3371745e88d4efbcec859a619 (127.28.112.1:42231), 1813e38e916647beb49a55a5bc392f01 (127.28.112.2:45051)
I20260809 07:53:51.203817 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:51.200622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:42231 (local address 127.28.112.3:37779)
0809 07:53:51.200743 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:51.200757 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:51.200845 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:51.201099 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:51.201103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:51.201123 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:51.201237 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:51.201243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.201972 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.201976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:51.202773 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:51.202782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.203407 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.203410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:51.203511 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.203535 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 07:53:51.203601 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 07:53:51.203687 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 07:53:51.203817 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:51.200676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:37779 (local address 127.28.112.1:42231)
0809 07:53:51.200803 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:51.200807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:51.200824 (+ 17us) server_negotiation.cc:407] Connection header received
0809 07:53:51.200933 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:51.200935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:51.200986 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:51.201064 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:51.201344 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.201845 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.202898 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.203297 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.203406 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.203460 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:51.203625 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 07:53:51.203695 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 07:53:51.204530 8254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda9e05b4d014830bd96c002d5ea6108" candidate_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f286b3371745e88d4efbcec859a619" is_pre_election: true
I20260809 07:53:51.204634 8781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:51.200682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:38595 (local address 127.28.112.2:45051)
0809 07:53:51.200803 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:51.200807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:51.201350 (+ 543us) server_negotiation.cc:407] Connection header received
0809 07:53:51.201472 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:51.201474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:51.201515 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:51.201576 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:51.202220 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.202705 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.203670 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.204133 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.204250 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.204291 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:51.204484 (+ 193us) server_negotiation.cc:299] Negotiation successful
0809 07:53:51.204540 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 07:53:51.204687 8795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:51.200622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:45051 (local address 127.28.112.3:38595)
0809 07:53:51.201252 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:51.201264 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:51.201361 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:51.201600 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:51.201603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:51.201619 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:51.202100 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:51.202108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.202804 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.202808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:51.203550 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:51.203558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.204269 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.204274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:51.204368 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.204393 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 07:53:51.204468 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 07:53:51.204533 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":280,"thread_start_us":125,"threads_started":1}
I20260809 07:53:51.204888 8254 raft_consensus.cc:2454] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ef4fcc7c19e4c7ca825f7930d1ae276 in term 0.
I20260809 07:53:51.205317 8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda9e05b4d014830bd96c002d5ea6108" candidate_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1813e38e916647beb49a55a5bc392f01" is_pre_election: true
I20260809 07:53:51.205394 8577 leader_election.cc:302] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ef4fcc7c19e4c7ca825f7930d1ae276, a3f286b3371745e88d4efbcec859a619; no voters:
I20260809 07:53:51.205585 8469 raft_consensus.cc:2454] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ef4fcc7c19e4c7ca825f7930d1ae276 in term 0.
I20260809 07:53:51.205675 8789 raft_consensus.cc:2781] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:53:51.205770 8789 raft_consensus.cc:482] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:53:51.205817 8789 raft_consensus.cc:3037] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:51.206881 8789 raft_consensus.cc:504] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.207216 8789 leader_election.cc:288] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [CANDIDATE]: Term 1 election: Requested vote from peers a3f286b3371745e88d4efbcec859a619 (127.28.112.1:42231), 1813e38e916647beb49a55a5bc392f01 (127.28.112.2:45051)
I20260809 07:53:51.207567 8254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda9e05b4d014830bd96c002d5ea6108" candidate_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f286b3371745e88d4efbcec859a619"
I20260809 07:53:51.207666 8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda9e05b4d014830bd96c002d5ea6108" candidate_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1813e38e916647beb49a55a5bc392f01"
I20260809 07:53:51.207697 8254 raft_consensus.cc:3037] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:51.207783 8469 raft_consensus.cc:3037] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:51.208956 8469 raft_consensus.cc:2454] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ef4fcc7c19e4c7ca825f7930d1ae276 in term 1.
I20260809 07:53:51.208956 8254 raft_consensus.cc:2454] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ef4fcc7c19e4c7ca825f7930d1ae276 in term 1.
I20260809 07:53:51.209296 8576 leader_election.cc:302] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1813e38e916647beb49a55a5bc392f01, 5ef4fcc7c19e4c7ca825f7930d1ae276; no voters:
I20260809 07:53:51.209539 8789 raft_consensus.cc:2781] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:51.209722 8789 raft_consensus.cc:686] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 LEADER]: Becoming Leader. State: Replica: 5ef4fcc7c19e4c7ca825f7930d1ae276, State: Running, Role: LEADER
I20260809 07:53:51.210064 8789 consensus_queue.cc:237] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.213094 8025 catalog_manager.cc:5495] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ef4fcc7c19e4c7ca825f7930d1ae276 (127.28.112.3). New cstate: current_term: 1 leader_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } health_report { overall_health: HEALTHY } } }
I20260809 07:53:51.257284 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:51.254310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.28.112.3:39831)
0809 07:53:51.254450 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:51.254454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:51.254469 (+ 15us) server_negotiation.cc:407] Connection header received
0809 07:53:51.254537 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:51.254539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:51.254591 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:51.254672 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:51.254850 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.255331 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.256103 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.256309 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.256383 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.256564 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:51.256977 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:51.257058 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:51.257101 (+ 43us) server_negotiation.cc:299] Negotiation successful
0809 07:53:51.257149 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 07:53:51.262737 8469 raft_consensus.cc:1261] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01 [term 1 FOLLOWER]: Refusing update from remote peer 5ef4fcc7c19e4c7ca825f7930d1ae276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:51.262737 8254 raft_consensus.cc:1261] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619 [term 1 FOLLOWER]: Refusing update from remote peer 5ef4fcc7c19e4c7ca825f7930d1ae276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:51.263398 8796 consensus_queue.cc:1035] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:51.263569 8789 consensus_queue.cc:1035] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:51.268963 8799 mvcc.cc:205] Tried to move back new op lower bound from 7316529280045588480 to 7316529279838994432. Current Snapshot: MvccSnapshot[applied={T|T < 7316529280045588480}]
I20260809 07:53:51.269982 8801 mvcc.cc:205] Tried to move back new op lower bound from 7316529280045588480 to 7316529279838994432. Current Snapshot: MvccSnapshot[applied={T|T < 7316529280045588480}]
I20260809 07:53:51.278230 8025 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:51.278398 8025 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:51.280649 8025 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b: Sending DeleteTablet for 3 replicas of tablet cda9e05b4d014830bd96c002d5ea6108
I20260809 07:53:51.281399 8440 tablet_service.cc:1514] Processing DeleteTablet for tablet cda9e05b4d014830bd96c002d5ea6108 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:51 UTC) from {username='oryx', principal='oryx/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:59868
I20260809 07:53:51.281435 8223 tablet_service.cc:1514] Processing DeleteTablet for tablet cda9e05b4d014830bd96c002d5ea6108 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:51 UTC) from {username='oryx', principal='oryx/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:50338
I20260809 07:53:51.281534 8641 tablet_service.cc:1514] Processing DeleteTablet for tablet cda9e05b4d014830bd96c002d5ea6108 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:51 UTC) from {username='oryx', principal='oryx/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:47000
I20260809 07:53:51.281936 8815 tablet_replica.cc:330] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276: stopping tablet replica
I20260809 07:53:51.282081 8813 tablet_replica.cc:330] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01: stopping tablet replica
I20260809 07:53:51.282274 8815 raft_consensus.cc:2229] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:53:51.282351 8813 raft_consensus.cc:2229] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:51.282610 8815 raft_consensus.cc:2258] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:51.282627 8813 raft_consensus.cc:2258] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:51.282713 8025 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 07:53:51.282968 8814 tablet_replica.cc:330] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619: stopping tablet replica
W20260809 07:53:51.283097 8025 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:53:51.283232 8814 raft_consensus.cc:2229] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:51.283489 8814 raft_consensus.cc:2258] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:51.284312 8815 ts_tablet_manager.cc:1905] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:51.285123 8813 ts_tablet_manager.cc:1905] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:51.287081 8815 ts_tablet_manager.cc:1918] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:53:51.287180 8815 log.cc:1196] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/cda9e05b4d014830bd96c002d5ea6108
I20260809 07:53:51.287505 8815 ts_tablet_manager.cc:1939] T cda9e05b4d014830bd96c002d5ea6108 P 5ef4fcc7c19e4c7ca825f7930d1ae276: Deleting consensus metadata
I20260809 07:53:51.287819 8813 ts_tablet_manager.cc:1918] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:53:51.287937 8813 log.cc:1196] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/cda9e05b4d014830bd96c002d5ea6108
I20260809 07:53:51.288240 8813 ts_tablet_manager.cc:1939] T cda9e05b4d014830bd96c002d5ea6108 P 1813e38e916647beb49a55a5bc392f01: Deleting consensus metadata
I20260809 07:53:51.288591 8006 catalog_manager.cc:4841] TS 5ef4fcc7c19e4c7ca825f7930d1ae276 (127.28.112.3:39831): tablet cda9e05b4d014830bd96c002d5ea6108 (table test-table [id=e052ef6be1a44db1959ad2c44c38ccba]) successfully deleted
I20260809 07:53:51.289091 8440 tablet_service.cc:1467] Processing CreateTablet for tablet 8962553192ef45668a477775112cda3f (DEFAULT_TABLE table=test-table [id=dd7cb2d107bc4529b3db9cc912f33ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:51.289091 8641 tablet_service.cc:1467] Processing CreateTablet for tablet 8962553192ef45668a477775112cda3f (DEFAULT_TABLE table=test-table [id=dd7cb2d107bc4529b3db9cc912f33ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:51.289332 8440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8962553192ef45668a477775112cda3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:51.289332 8641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8962553192ef45668a477775112cda3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:51.290109 8814 ts_tablet_manager.cc:1905] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:51.291363 8785 tablet_bootstrap.cc:492] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01: Bootstrap starting.
I20260809 07:53:51.291363 8782 tablet_bootstrap.cc:492] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276: Bootstrap starting.
I20260809 07:53:51.292259 8785 tablet_bootstrap.cc:654] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:51.292259 8782 tablet_bootstrap.cc:654] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:51.292784 8814 ts_tablet_manager.cc:1918] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:53:51.292898 8814 log.cc:1196] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/wal/wals/cda9e05b4d014830bd96c002d5ea6108
I20260809 07:53:51.293196 8814 ts_tablet_manager.cc:1939] T cda9e05b4d014830bd96c002d5ea6108 P a3f286b3371745e88d4efbcec859a619: Deleting consensus metadata
I20260809 07:53:51.293227 8004 catalog_manager.cc:4841] TS 1813e38e916647beb49a55a5bc392f01 (127.28.112.2:45051): tablet cda9e05b4d014830bd96c002d5ea6108 (table test-table [id=e052ef6be1a44db1959ad2c44c38ccba]) successfully deleted
I20260809 07:53:51.294042 8785 tablet_bootstrap.cc:492] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01: No bootstrap required, opened a new log
I20260809 07:53:51.294129 8785 ts_tablet_manager.cc:1397] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260809 07:53:51.294195 8223 tablet_service.cc:1467] Processing CreateTablet for tablet 8962553192ef45668a477775112cda3f (DEFAULT_TABLE table=test-table [id=dd7cb2d107bc4529b3db9cc912f33ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:51.294459 8223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8962553192ef45668a477775112cda3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:51.294570 8785 raft_consensus.cc:348] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.294677 8785 raft_consensus.cc:374] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:51.294718 8785 raft_consensus.cc:729] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1813e38e916647beb49a55a5bc392f01, State: Initialized, Role: FOLLOWER
I20260809 07:53:51.294863 8785 consensus_queue.cc:260] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.295122 8785 ts_tablet_manager.cc:1428] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:51.296528 8783 tablet_bootstrap.cc:492] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619: Bootstrap starting.
I20260809 07:53:51.296609 8005 catalog_manager.cc:4841] TS a3f286b3371745e88d4efbcec859a619 (127.28.112.1:42231): tablet cda9e05b4d014830bd96c002d5ea6108 (table test-table [id=e052ef6be1a44db1959ad2c44c38ccba]) successfully deleted
I20260809 07:53:51.297394 8783 tablet_bootstrap.cc:654] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:51.297561 8782 tablet_bootstrap.cc:492] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276: No bootstrap required, opened a new log
I20260809 07:53:51.297638 8782 ts_tablet_manager.cc:1397] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260809 07:53:51.298112 8782 raft_consensus.cc:348] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.298223 8782 raft_consensus.cc:374] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:51.298255 8782 raft_consensus.cc:729] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ef4fcc7c19e4c7ca825f7930d1ae276, State: Initialized, Role: FOLLOWER
I20260809 07:53:51.298365 8782 consensus_queue.cc:260] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.298472 8783 tablet_bootstrap.cc:492] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619: No bootstrap required, opened a new log
I20260809 07:53:51.298542 8783 ts_tablet_manager.cc:1397] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:51.298623 8782 ts_tablet_manager.cc:1428] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:51.299075 8783 raft_consensus.cc:348] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.299180 8783 raft_consensus.cc:374] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:51.299214 8783 raft_consensus.cc:729] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f286b3371745e88d4efbcec859a619, State: Initialized, Role: FOLLOWER
I20260809 07:53:51.299336 8783 consensus_queue.cc:260] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.299574 8783 ts_tablet_manager.cc:1428] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 07:53:51.435812 8792 raft_consensus.cc:482] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:51.435945 8792 raft_consensus.cc:504] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.436933 8792 leader_election.cc:288] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1813e38e916647beb49a55a5bc392f01 (127.28.112.2:45051), 5ef4fcc7c19e4c7ca825f7930d1ae276 (127.28.112.3:39831)
I20260809 07:53:51.442149 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:51.437333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:42091 (local address 127.28.112.3:39831)
0809 07:53:51.437467 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:51.437472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:51.437631 (+ 159us) server_negotiation.cc:407] Connection header received
0809 07:53:51.437732 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:51.437735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:51.437779 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:51.437858 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:51.438224 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.439007 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.440793 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.441410 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.441528 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.441581 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:51.441912 (+ 331us) server_negotiation.cc:299] Negotiation successful
0809 07:53:51.442033 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 07:53:51.442149 8781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:51.437269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:42193 (local address 127.28.112.2:45051)
0809 07:53:51.437487 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:51.437492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:51.437561 (+ 69us) server_negotiation.cc:407] Connection header received
0809 07:53:51.437721 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:51.437724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:51.437779 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:51.437854 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:51.438224 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.439007 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.440819 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.441410 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.441533 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.441581 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:51.441912 (+ 331us) server_negotiation.cc:299] Negotiation successful
0809 07:53:51.442023 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260809 07:53:51.442167 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:51.437173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:45051 (local address 127.28.112.1:42193)
0809 07:53:51.437465 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:51.437481 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:51.437604 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:51.437877 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:51.437880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:51.437912 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:51.438108 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:51.438115 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.439109 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.439112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:51.440612 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:51.440625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.441532 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.441535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:51.441704 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.441785 (+ 81us) client_negotiation.cc:769] Sending connection context
0809 07:53:51.441882 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 07:53:51.441983 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":177,"spinlock_wait_cycles":12160,"thread_start_us":80,"threads_started":1}
I20260809 07:53:51.442167 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:51.437198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:39831 (local address 127.28.112.1:42091)
0809 07:53:51.437554 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:51.437565 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:51.437647 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:51.437877 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:51.437880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:51.437906 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:51.438108 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:51.438115 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.439131 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.439143 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:51.440612 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:51.440625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.441529 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.441533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:51.441705 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.441767 (+ 62us) client_negotiation.cc:769] Sending connection context
0809 07:53:51.441882 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 07:53:51.442005 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":241,"spinlock_wait_cycles":1280,"thread_start_us":111,"threads_started":1}
I20260809 07:53:51.442905 8674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8962553192ef45668a477775112cda3f" candidate_uuid: "a3f286b3371745e88d4efbcec859a619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" is_pre_election: true
I20260809 07:53:51.443117 8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8962553192ef45668a477775112cda3f" candidate_uuid: "a3f286b3371745e88d4efbcec859a619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1813e38e916647beb49a55a5bc392f01" is_pre_election: true
I20260809 07:53:51.443173 8674 raft_consensus.cc:2454] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3f286b3371745e88d4efbcec859a619 in term 0.
I20260809 07:53:51.443274 8469 raft_consensus.cc:2454] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3f286b3371745e88d4efbcec859a619 in term 0.
I20260809 07:53:51.443578 8152 leader_election.cc:302] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ef4fcc7c19e4c7ca825f7930d1ae276, a3f286b3371745e88d4efbcec859a619; no voters:
I20260809 07:53:51.443843 8792 raft_consensus.cc:2781] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:53:51.443902 8792 raft_consensus.cc:482] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:53:51.443959 8792 raft_consensus.cc:3037] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:51.444941 8792 raft_consensus.cc:504] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.445266 8792 leader_election.cc:288] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [CANDIDATE]: Term 1 election: Requested vote from peers 1813e38e916647beb49a55a5bc392f01 (127.28.112.2:45051), 5ef4fcc7c19e4c7ca825f7930d1ae276 (127.28.112.3:39831)
I20260809 07:53:51.445597 8674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8962553192ef45668a477775112cda3f" candidate_uuid: "a3f286b3371745e88d4efbcec859a619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276"
I20260809 07:53:51.445597 8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8962553192ef45668a477775112cda3f" candidate_uuid: "a3f286b3371745e88d4efbcec859a619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1813e38e916647beb49a55a5bc392f01"
I20260809 07:53:51.445721 8674 raft_consensus.cc:3037] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:51.445721 8469 raft_consensus.cc:3037] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:51.446810 8469 raft_consensus.cc:2454] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3f286b3371745e88d4efbcec859a619 in term 1.
I20260809 07:53:51.446810 8674 raft_consensus.cc:2454] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3f286b3371745e88d4efbcec859a619 in term 1.
I20260809 07:53:51.447093 8149 leader_election.cc:302] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1813e38e916647beb49a55a5bc392f01, 5ef4fcc7c19e4c7ca825f7930d1ae276, a3f286b3371745e88d4efbcec859a619; no voters:
I20260809 07:53:51.447311 8792 raft_consensus.cc:2781] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:51.447474 8792 raft_consensus.cc:686] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [term 1 LEADER]: Becoming Leader. State: Replica: a3f286b3371745e88d4efbcec859a619, State: Running, Role: LEADER
I20260809 07:53:51.447706 8792 consensus_queue.cc:237] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.449801 8025 catalog_manager.cc:5495] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3f286b3371745e88d4efbcec859a619 (127.28.112.1). New cstate: current_term: 1 leader_uuid: "a3f286b3371745e88d4efbcec859a619" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } health_report { overall_health: UNKNOWN } } }
Aug 09 07:53:51 dist-test-slave-vpvm krb5kdc[7959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262030, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.62@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM
I20260809 07:53:51.528121 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:51.519688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:45347 (local address 127.0.0.1:46074)
0809 07:53:51.519981 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:51.519996 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:51.520131 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:51.520379 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:51.520381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:51.520797 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:51.520896 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:51.520903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.521650 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.521654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:51.522269 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:51.522279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.522698 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.522701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:51.522862 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.523401 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:51.523413 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:51.525010 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:51.526614 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:51.526623 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:51.526631 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:51.526854 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:51.527107 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:51.527110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:51.527112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:51.527187 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:51.527541 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:51.527548 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:51.527656 (+ 108us) client_negotiation.cc:769] Sending connection context
0809 07:53:51.527750 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 07:53:51.527907 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":65,"thread_start_us":87,"threads_started":1}
I20260809 07:53:51.528121 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:51.519759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46074 (local address 127.28.112.62:45347)
0809 07:53:51.519996 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:51.520005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:51.520076 (+ 71us) server_negotiation.cc:407] Connection header received
0809 07:53:51.520229 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:51.520232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:51.520272 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:51.520345 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:51.521017 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.521547 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.522379 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.522594 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.522691 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.525140 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:51.525144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:51.525146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:51.525168 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:51.526507 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:51.526940 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:51.526944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:51.526945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:51.527028 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:51.527261 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:51.527265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:51.527266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:51.527437 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:51.527519 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:51.527808 (+ 289us) server_negotiation.cc:299] Negotiation successful
0809 07:53:51.527959 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":186,"thread_start_us":105,"threads_started":1}
I20260809 07:53:51.532070 8025 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:46074:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 07:53:51.532732 8025 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:53:51.537611 8440 tablet_service.cc:1467] Processing CreateTablet for tablet 54114b7e60164779bbd34ba803b08f3d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1dec526f5ce04a04874c34287795d16f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 07:53:51.537653 8223 tablet_service.cc:1467] Processing CreateTablet for tablet 54114b7e60164779bbd34ba803b08f3d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1dec526f5ce04a04874c34287795d16f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 07:53:51.537710 8641 tablet_service.cc:1467] Processing CreateTablet for tablet 54114b7e60164779bbd34ba803b08f3d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1dec526f5ce04a04874c34287795d16f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 07:53:51.538061 8223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54114b7e60164779bbd34ba803b08f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:51.538061 8641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54114b7e60164779bbd34ba803b08f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:51.538061 8440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54114b7e60164779bbd34ba803b08f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:51.540577 8783 tablet_bootstrap.cc:492] T 54114b7e60164779bbd34ba803b08f3d P a3f286b3371745e88d4efbcec859a619: Bootstrap starting.
I20260809 07:53:51.540736 8782 tablet_bootstrap.cc:492] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: Bootstrap starting.
I20260809 07:53:51.540808 8785 tablet_bootstrap.cc:492] T 54114b7e60164779bbd34ba803b08f3d P 1813e38e916647beb49a55a5bc392f01: Bootstrap starting.
I20260809 07:53:51.541582 8783 tablet_bootstrap.cc:654] T 54114b7e60164779bbd34ba803b08f3d P a3f286b3371745e88d4efbcec859a619: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:51.541584 8782 tablet_bootstrap.cc:654] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:51.541767 8785 tablet_bootstrap.cc:654] T 54114b7e60164779bbd34ba803b08f3d P 1813e38e916647beb49a55a5bc392f01: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:51.542886 8782 tablet_bootstrap.cc:492] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: No bootstrap required, opened a new log
I20260809 07:53:51.542979 8782 ts_tablet_manager.cc:1397] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260809 07:53:51.543201 8785 tablet_bootstrap.cc:492] T 54114b7e60164779bbd34ba803b08f3d P 1813e38e916647beb49a55a5bc392f01: No bootstrap required, opened a new log
I20260809 07:53:51.543287 8785 ts_tablet_manager.cc:1397] T 54114b7e60164779bbd34ba803b08f3d P 1813e38e916647beb49a55a5bc392f01: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260809 07:53:51.543411 8782 raft_consensus.cc:348] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.543540 8782 raft_consensus.cc:374] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:51.543581 8782 raft_consensus.cc:729] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ef4fcc7c19e4c7ca825f7930d1ae276, State: Initialized, Role: FOLLOWER
I20260809 07:53:51.543705 8785 raft_consensus.cc:348] T 54114b7e60164779bbd34ba803b08f3d P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.543711 8782 consensus_queue.cc:260] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.543807 8785 raft_consensus.cc:374] T 54114b7e60164779bbd34ba803b08f3d P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:51.543843 8785 raft_consensus.cc:729] T 54114b7e60164779bbd34ba803b08f3d P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1813e38e916647beb49a55a5bc392f01, State: Initialized, Role: FOLLOWER
I20260809 07:53:51.543993 8782 ts_tablet_manager.cc:1428] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 07:53:51.543972 8785 consensus_queue.cc:260] T 54114b7e60164779bbd34ba803b08f3d P 1813e38e916647beb49a55a5bc392f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.544225 8785 ts_tablet_manager.cc:1428] T 54114b7e60164779bbd34ba803b08f3d P 1813e38e916647beb49a55a5bc392f01: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 07:53:51.544226 8793 tablet_replica.cc:439] T 54114b7e60164779bbd34ba803b08f3d P 1813e38e916647beb49a55a5bc392f01: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } } }
I20260809 07:53:51.544387 8793 tablet_replica.cc:442] T 54114b7e60164779bbd34ba803b08f3d P 1813e38e916647beb49a55a5bc392f01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:51.544351 8806 tablet_replica.cc:439] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } } }
I20260809 07:53:51.544476 8806 tablet_replica.cc:442] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:51.544515 8783 tablet_bootstrap.cc:492] T 54114b7e60164779bbd34ba803b08f3d P a3f286b3371745e88d4efbcec859a619: No bootstrap required, opened a new log
I20260809 07:53:51.544598 8783 ts_tablet_manager.cc:1397] T 54114b7e60164779bbd34ba803b08f3d P a3f286b3371745e88d4efbcec859a619: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260809 07:53:51.545037 8783 raft_consensus.cc:348] T 54114b7e60164779bbd34ba803b08f3d P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.545152 8783 raft_consensus.cc:374] T 54114b7e60164779bbd34ba803b08f3d P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:51.545190 8783 raft_consensus.cc:729] T 54114b7e60164779bbd34ba803b08f3d P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f286b3371745e88d4efbcec859a619, State: Initialized, Role: FOLLOWER
I20260809 07:53:51.545311 8783 consensus_queue.cc:260] T 54114b7e60164779bbd34ba803b08f3d P a3f286b3371745e88d4efbcec859a619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.545579 8783 ts_tablet_manager.cc:1428] T 54114b7e60164779bbd34ba803b08f3d P a3f286b3371745e88d4efbcec859a619: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 07:53:51.546097 8792 tablet_replica.cc:439] T 54114b7e60164779bbd34ba803b08f3d P a3f286b3371745e88d4efbcec859a619: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } } }
I20260809 07:53:51.546231 8792 tablet_replica.cc:442] T 54114b7e60164779bbd34ba803b08f3d P a3f286b3371745e88d4efbcec859a619: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:51.729476 8806 raft_consensus.cc:482] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:51.729629 8806 raft_consensus.cc:504] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.730130 8806 leader_election.cc:288] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3f286b3371745e88d4efbcec859a619 (127.28.112.1:42231), 1813e38e916647beb49a55a5bc392f01 (127.28.112.2:45051)
I20260809 07:53:51.730582 8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54114b7e60164779bbd34ba803b08f3d" candidate_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1813e38e916647beb49a55a5bc392f01" is_pre_election: true
I20260809 07:53:51.730582 8254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54114b7e60164779bbd34ba803b08f3d" candidate_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f286b3371745e88d4efbcec859a619" is_pre_election: true
I20260809 07:53:51.730744 8254 raft_consensus.cc:2454] T 54114b7e60164779bbd34ba803b08f3d P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ef4fcc7c19e4c7ca825f7930d1ae276 in term 0.
I20260809 07:53:51.730744 8469 raft_consensus.cc:2454] T 54114b7e60164779bbd34ba803b08f3d P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ef4fcc7c19e4c7ca825f7930d1ae276 in term 0.
I20260809 07:53:51.731062 8577 leader_election.cc:302] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1813e38e916647beb49a55a5bc392f01, 5ef4fcc7c19e4c7ca825f7930d1ae276, a3f286b3371745e88d4efbcec859a619; no voters:
I20260809 07:53:51.731280 8806 raft_consensus.cc:2781] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:53:51.731351 8806 raft_consensus.cc:482] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:53:51.731380 8806 raft_consensus.cc:3037] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:51.732410 8806 raft_consensus.cc:504] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.732740 8806 leader_election.cc:288] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [CANDIDATE]: Term 1 election: Requested vote from peers a3f286b3371745e88d4efbcec859a619 (127.28.112.1:42231), 1813e38e916647beb49a55a5bc392f01 (127.28.112.2:45051)
I20260809 07:53:51.733044 8254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54114b7e60164779bbd34ba803b08f3d" candidate_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f286b3371745e88d4efbcec859a619"
I20260809 07:53:51.733062 8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54114b7e60164779bbd34ba803b08f3d" candidate_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1813e38e916647beb49a55a5bc392f01"
I20260809 07:53:51.733152 8254 raft_consensus.cc:3037] T 54114b7e60164779bbd34ba803b08f3d P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:51.733181 8469 raft_consensus.cc:3037] T 54114b7e60164779bbd34ba803b08f3d P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:51.734045 8469 raft_consensus.cc:2454] T 54114b7e60164779bbd34ba803b08f3d P 1813e38e916647beb49a55a5bc392f01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ef4fcc7c19e4c7ca825f7930d1ae276 in term 1.
I20260809 07:53:51.734068 8254 raft_consensus.cc:2454] T 54114b7e60164779bbd34ba803b08f3d P a3f286b3371745e88d4efbcec859a619 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ef4fcc7c19e4c7ca825f7930d1ae276 in term 1.
I20260809 07:53:51.734309 8576 leader_election.cc:302] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1813e38e916647beb49a55a5bc392f01, 5ef4fcc7c19e4c7ca825f7930d1ae276; no voters:
I20260809 07:53:51.734478 8806 raft_consensus.cc:2781] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:51.734580 8806 raft_consensus.cc:686] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 LEADER]: Becoming Leader. State: Replica: 5ef4fcc7c19e4c7ca825f7930d1ae276, State: Running, Role: LEADER
I20260809 07:53:51.734719 8806 consensus_queue.cc:237] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.735340 8789 tablet_replica.cc:439] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: TxnStatusTablet state changed. Reason: New leader 5ef4fcc7c19e4c7ca825f7930d1ae276. Latest consensus state: current_term: 1 leader_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } } }
I20260809 07:53:51.735446 8789 tablet_replica.cc:442] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: This TxnStatusTablet replica's current role is: LEADER
I20260809 07:53:51.735661 8847 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 07:53:51.735822 8847 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 07:53:51.736433 8025 catalog_manager.cc:5495] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ef4fcc7c19e4c7ca825f7930d1ae276 (127.28.112.3). New cstate: current_term: 1 leader_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } health_report { overall_health: HEALTHY } } }
I20260809 07:53:51.779458 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:51.776335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:39831 (local address 127.0.0.1:47032)
0809 07:53:51.776492 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:51.776508 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:51.776617 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:51.776862 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:51.776865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:51.776872 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 07:53:51.776976 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:51.776981 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.777676 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.777680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:51.778332 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:51.778341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.778762 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.778765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:51.778853 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.778899 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 07:53:51.779172 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 07:53:51.779187 (+ 15us) client_negotiation.cc:769] Sending connection context
0809 07:53:51.779247 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 07:53:51.779328 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 07:53:51.779449 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:51.776475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47032 (local address 127.28.112.3:39831)
0809 07:53:51.776597 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:51.776601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:51.776616 (+ 15us) server_negotiation.cc:407] Connection header received
0809 07:53:51.776722 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:51.776725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:51.776773 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:51.776833 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:51.777071 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.777573 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.778442 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.778664 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.778753 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.779025 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:51.779098 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:51.779158 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:51.779259 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 07:53:51.779309 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 07:53:51.782248 8469 raft_consensus.cc:1261] T 54114b7e60164779bbd34ba803b08f3d P 1813e38e916647beb49a55a5bc392f01 [term 1 FOLLOWER]: Refusing update from remote peer 5ef4fcc7c19e4c7ca825f7930d1ae276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:51.782404 8254 raft_consensus.cc:1261] T 54114b7e60164779bbd34ba803b08f3d P a3f286b3371745e88d4efbcec859a619 [term 1 FOLLOWER]: Refusing update from remote peer 5ef4fcc7c19e4c7ca825f7930d1ae276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:51.782743 8789 consensus_queue.cc:1035] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:51.782934 8806 consensus_queue.cc:1035] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:51.784281 8792 tablet_replica.cc:439] T 54114b7e60164779bbd34ba803b08f3d P a3f286b3371745e88d4efbcec859a619: TxnStatusTablet state changed. Reason: New leader 5ef4fcc7c19e4c7ca825f7930d1ae276. Latest consensus state: current_term: 1 leader_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } } }
I20260809 07:53:51.784392 8792 tablet_replica.cc:442] T 54114b7e60164779bbd34ba803b08f3d P a3f286b3371745e88d4efbcec859a619: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:51.784363 8793 tablet_replica.cc:439] T 54114b7e60164779bbd34ba803b08f3d P 1813e38e916647beb49a55a5bc392f01: TxnStatusTablet state changed. Reason: New leader 5ef4fcc7c19e4c7ca825f7930d1ae276. Latest consensus state: current_term: 1 leader_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } } }
I20260809 07:53:51.784525 8793 tablet_replica.cc:442] T 54114b7e60164779bbd34ba803b08f3d P 1813e38e916647beb49a55a5bc392f01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:51.787575 8793 tablet_replica.cc:439] T 54114b7e60164779bbd34ba803b08f3d P 1813e38e916647beb49a55a5bc392f01: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } } }
I20260809 07:53:51.787691 8793 tablet_replica.cc:442] T 54114b7e60164779bbd34ba803b08f3d P 1813e38e916647beb49a55a5bc392f01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:51.787988 8806 tablet_replica.cc:439] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } } }
I20260809 07:53:51.788105 8806 tablet_replica.cc:442] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: This TxnStatusTablet replica's current role is: LEADER
I20260809 07:53:51.788228 8857 mvcc.cc:205] Tried to move back new op lower bound from 7316529282175787008 to 7316529281986514944. Current Snapshot: MvccSnapshot[applied={T|T < 7316529282175787008}]
I20260809 07:53:51.790088 8792 tablet_replica.cc:439] T 54114b7e60164779bbd34ba803b08f3d P a3f286b3371745e88d4efbcec859a619: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } } }
I20260809 07:53:51.790198 8792 tablet_replica.cc:442] T 54114b7e60164779bbd34ba803b08f3d P a3f286b3371745e88d4efbcec859a619: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 07:53:51.790642 8864 tablet_replica.cc:439] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } } }
I20260809 07:53:51.790742 8864 tablet_replica.cc:442] T 54114b7e60164779bbd34ba803b08f3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: This TxnStatusTablet replica's current role is: LEADER
I20260809 07:53:51.797693 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:51.793616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.28.112.1:42231)
0809 07:53:51.793851 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:51.793857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:51.793870 (+ 13us) server_negotiation.cc:407] Connection header received
0809 07:53:51.794078 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:51.794080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:51.794131 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:51.794224 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:51.794538 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.795065 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.796281 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.796523 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.796603 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.796879 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:51.797320 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:51.797419 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:51.797509 (+ 90us) server_negotiation.cc:299] Negotiation successful
0809 07:53:51.797561 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":106,"threads_started":1}
I20260809 07:53:51.808038 8885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:51.804063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:39831 (local address 127.28.112.1:56573)
0809 07:53:51.804395 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:51.804407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:51.804491 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:51.804711 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:51.804713 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:51.804718 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 07:53:51.804800 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:51.804804 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.805725 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.805728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:51.806755 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:51.806765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.807344 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.807347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:51.807419 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.807464 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 07:53:51.807756 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 07:53:51.807764 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 07:53:51.807830 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 07:53:51.807908 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":70,"threads_started":1}
I20260809 07:53:51.808058 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:51.804158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:56573 (local address 127.28.112.3:39831)
0809 07:53:51.804270 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:51.804274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:51.804473 (+ 199us) server_negotiation.cc:407] Connection header received
0809 07:53:51.804569 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:51.804572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:51.804626 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:51.804711 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:51.804879 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.805630 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.806863 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.807211 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.807337 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.807582 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:51.807664 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:51.807760 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:51.807860 (+ 100us) server_negotiation.cc:299] Negotiation successful
0809 07:53:51.807918 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 07:53:51.814845 8469 raft_consensus.cc:1261] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01 [term 1 FOLLOWER]: Refusing update from remote peer a3f286b3371745e88d4efbcec859a619: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:51.815106 8674 raft_consensus.cc:1261] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 FOLLOWER]: Refusing update from remote peer a3f286b3371745e88d4efbcec859a619: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:51.815486 8828 consensus_queue.cc:1035] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:51.815675 8792 consensus_queue.cc:1035] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:51.838205 8891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:51.834148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:42231 (local address 127.28.112.3:42335)
0809 07:53:51.834518 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:51.834534 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:51.834731 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:51.834996 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:51.834999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:51.835008 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 07:53:51.835111 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:51.835116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.836087 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.836090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:51.836890 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:51.836902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.837459 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.837463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:51.837554 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.837608 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 07:53:51.837898 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 07:53:51.837903 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 07:53:51.837993 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 07:53:51.838073 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":132,"threads_started":1}
I20260809 07:53:51.838223 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:51.834198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:42335 (local address 127.28.112.1:42231)
0809 07:53:51.834467 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:51.834471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:51.834616 (+ 145us) server_negotiation.cc:407] Connection header received
0809 07:53:51.834832 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:51.834835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:51.834889 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:51.834963 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:51.835217 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.835983 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.837009 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.837337 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.837460 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.837731 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:51.837816 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:51.837891 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:51.838008 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 07:53:51.838072 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260809 07:53:51.873490 8024 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:51.873668 8024 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:51.876468 8024 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b: Sending DeleteTablet for 3 replicas of tablet 8962553192ef45668a477775112cda3f
I20260809 07:53:51.877197 8641 tablet_service.cc:1514] Processing DeleteTablet for tablet 8962553192ef45668a477775112cda3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:51 UTC) from {username='oryx', principal='oryx/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:47000
I20260809 07:53:51.877193 8440 tablet_service.cc:1514] Processing DeleteTablet for tablet 8962553192ef45668a477775112cda3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:51 UTC) from {username='oryx', principal='oryx/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:59868
I20260809 07:53:51.877849 8223 tablet_service.cc:1514] Processing DeleteTablet for tablet 8962553192ef45668a477775112cda3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:51 UTC) from {username='oryx', principal='oryx/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:50338
I20260809 07:53:51.878273 8895 tablet_replica.cc:330] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01: stopping tablet replica
I20260809 07:53:51.878451 8895 raft_consensus.cc:2229] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:51.878626 8895 raft_consensus.cc:2258] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:51.879398 8895 ts_tablet_manager.cc:1905] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:51.882128 8897 tablet_replica.cc:330] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619: stopping tablet replica
I20260809 07:53:51.882282 8897 raft_consensus.cc:2229] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:53:51.882575 8895 ts_tablet_manager.cc:1918] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 07:53:51.882608 8897 raft_consensus.cc:2258] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:51.882727 8895 log.cc:1196] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/8962553192ef45668a477775112cda3f
I20260809 07:53:51.883055 8895 ts_tablet_manager.cc:1939] T 8962553192ef45668a477775112cda3f P 1813e38e916647beb49a55a5bc392f01: Deleting consensus metadata
I20260809 07:53:51.883832 8894 tablet_replica.cc:330] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276: stopping tablet replica
I20260809 07:53:51.883882 8004 catalog_manager.cc:4841] TS 1813e38e916647beb49a55a5bc392f01 (127.28.112.2:45051): tablet 8962553192ef45668a477775112cda3f (table test-table [id=dd7cb2d107bc4529b3db9cc912f33ee2]) successfully deleted
I20260809 07:53:51.884065 8894 raft_consensus.cc:2229] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:51.884222 8894 raft_consensus.cc:2258] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:51.884935 8897 ts_tablet_manager.cc:1905] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:51.885437 8894 ts_tablet_manager.cc:1905] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:51.890566 8897 ts_tablet_manager.cc:1918] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 07:53:51.891310 8894 ts_tablet_manager.cc:1918] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 07:53:51.891392 8894 log.cc:1196] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/8962553192ef45668a477775112cda3f
I20260809 07:53:51.891554 8897 log.cc:1196] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/wal/wals/8962553192ef45668a477775112cda3f
I20260809 07:53:51.891639 8894 ts_tablet_manager.cc:1939] T 8962553192ef45668a477775112cda3f P 5ef4fcc7c19e4c7ca825f7930d1ae276: Deleting consensus metadata
I20260809 07:53:51.892004 8897 ts_tablet_manager.cc:1939] T 8962553192ef45668a477775112cda3f P a3f286b3371745e88d4efbcec859a619: Deleting consensus metadata
I20260809 07:53:51.892354 8006 catalog_manager.cc:4841] TS 5ef4fcc7c19e4c7ca825f7930d1ae276 (127.28.112.3:39831): tablet 8962553192ef45668a477775112cda3f (table test-table [id=dd7cb2d107bc4529b3db9cc912f33ee2]) successfully deleted
I20260809 07:53:51.892879 8005 catalog_manager.cc:4841] TS a3f286b3371745e88d4efbcec859a619 (127.28.112.1:42231): tablet 8962553192ef45668a477775112cda3f (table test-table [id=dd7cb2d107bc4529b3db9cc912f33ee2]) successfully deleted
Aug 09 07:53:51 dist-test-slave-vpvm krb5kdc[7959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 07:53:51 dist-test-slave-vpvm krb5kdc[7959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:53:51 dist-test-slave-vpvm krb5kdc[7959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 07:53:51.908414 8840 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:51.901561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46086 (local address 127.28.112.62:45347)
0809 07:53:51.901711 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:51.901717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:51.901898 (+ 181us) server_negotiation.cc:407] Connection header received
0809 07:53:51.902034 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:51.902037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:51.902093 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:51.902191 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:51.902992 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.903520 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.904679 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.904990 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.905082 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.908286 (+ 3204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46086: BlockingRecv error: recv got EOF from 127.0.0.1:46086 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 09 07:53:51 dist-test-slave-vpvm krb5kdc[7959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM
I20260809 07:53:51.918061 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:51.909910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46100 (local address 127.28.112.62:45347)
0809 07:53:51.910040 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:51.910044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:51.910166 (+ 122us) server_negotiation.cc:407] Connection header received
0809 07:53:51.910282 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:51.910284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:51.910323 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:51.910377 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:51.911021 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.911493 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.912534 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:51.912742 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:51.912808 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:51.915123 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:51.915126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:51.915128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:51.915160 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:51.916402 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:51.916806 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:51.916809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:51.916810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:51.916934 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:51.917218 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:51.917221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:51.917223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:51.917393 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:51.917509 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:51.917750 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 07:53:51.917861 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 07:53:51.920516 8024 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:53:51.920975 8024 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:53:51.925733 8223 tablet_service.cc:1467] Processing CreateTablet for tablet a0b56fbef9a6438e8426df083e78d5d1 (DEFAULT_TABLE table=test-table [id=69afc1b6b8674de092cb074820c8bc99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:51.925863 8440 tablet_service.cc:1467] Processing CreateTablet for tablet a0b56fbef9a6438e8426df083e78d5d1 (DEFAULT_TABLE table=test-table [id=69afc1b6b8674de092cb074820c8bc99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:51.926074 8223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b56fbef9a6438e8426df083e78d5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:51.926222 8440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b56fbef9a6438e8426df083e78d5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:51.928107 8785 tablet_bootstrap.cc:492] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01: Bootstrap starting.
I20260809 07:53:51.928391 8783 tablet_bootstrap.cc:492] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619: Bootstrap starting.
I20260809 07:53:51.928900 8785 tablet_bootstrap.cc:654] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:51.929251 8783 tablet_bootstrap.cc:654] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:51.929927 8785 tablet_bootstrap.cc:492] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01: No bootstrap required, opened a new log
I20260809 07:53:51.930042 8785 ts_tablet_manager.cc:1397] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260809 07:53:51.930400 8785 raft_consensus.cc:348] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.930478 8785 raft_consensus.cc:374] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:51.930498 8785 raft_consensus.cc:729] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1813e38e916647beb49a55a5bc392f01, State: Initialized, Role: FOLLOWER
I20260809 07:53:51.930603 8785 consensus_queue.cc:260] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.930656 8641 tablet_service.cc:1467] Processing CreateTablet for tablet a0b56fbef9a6438e8426df083e78d5d1 (DEFAULT_TABLE table=test-table [id=69afc1b6b8674de092cb074820c8bc99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:51.930859 8785 ts_tablet_manager.cc:1428] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:51.930912 8641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b56fbef9a6438e8426df083e78d5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:51.933163 8782 tablet_bootstrap.cc:492] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276: Bootstrap starting.
I20260809 07:53:51.933821 8782 tablet_bootstrap.cc:654] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:51.934043 8783 tablet_bootstrap.cc:492] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619: No bootstrap required, opened a new log
I20260809 07:53:51.934132 8783 ts_tablet_manager.cc:1397] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260809 07:53:51.934573 8783 raft_consensus.cc:348] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.934685 8783 raft_consensus.cc:374] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:51.934723 8783 raft_consensus.cc:729] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f286b3371745e88d4efbcec859a619, State: Initialized, Role: FOLLOWER
I20260809 07:53:51.934841 8783 consensus_queue.cc:260] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.934976 8782 tablet_bootstrap.cc:492] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276: No bootstrap required, opened a new log
I20260809 07:53:51.935065 8782 ts_tablet_manager.cc:1397] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260809 07:53:51.935096 8783 ts_tablet_manager.cc:1428] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 07:53:51.935454 8782 raft_consensus.cc:348] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.935568 8782 raft_consensus.cc:374] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:51.935607 8782 raft_consensus.cc:729] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ef4fcc7c19e4c7ca825f7930d1ae276, State: Initialized, Role: FOLLOWER
I20260809 07:53:51.935745 8782 consensus_queue.cc:260] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:51.935992 8782 ts_tablet_manager.cc:1428] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 07:53:52.110294 8789 raft_consensus.cc:482] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:52.110435 8789 raft_consensus.cc:504] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:52.110873 8789 leader_election.cc:288] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3f286b3371745e88d4efbcec859a619 (127.28.112.1:42231), 1813e38e916647beb49a55a5bc392f01 (127.28.112.2:45051)
I20260809 07:53:52.111348 8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b56fbef9a6438e8426df083e78d5d1" candidate_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1813e38e916647beb49a55a5bc392f01" is_pre_election: true
I20260809 07:53:52.111382 8254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b56fbef9a6438e8426df083e78d5d1" candidate_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f286b3371745e88d4efbcec859a619" is_pre_election: true
I20260809 07:53:52.111506 8469 raft_consensus.cc:2454] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ef4fcc7c19e4c7ca825f7930d1ae276 in term 0.
I20260809 07:53:52.111521 8254 raft_consensus.cc:2454] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ef4fcc7c19e4c7ca825f7930d1ae276 in term 0.
I20260809 07:53:52.111804 8576 leader_election.cc:302] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1813e38e916647beb49a55a5bc392f01, 5ef4fcc7c19e4c7ca825f7930d1ae276; no voters:
I20260809 07:53:52.111982 8789 raft_consensus.cc:2781] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:53:52.112043 8789 raft_consensus.cc:482] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:53:52.112089 8789 raft_consensus.cc:3037] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:52.113070 8789 raft_consensus.cc:504] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:52.113399 8789 leader_election.cc:288] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [CANDIDATE]: Term 1 election: Requested vote from peers a3f286b3371745e88d4efbcec859a619 (127.28.112.1:42231), 1813e38e916647beb49a55a5bc392f01 (127.28.112.2:45051)
I20260809 07:53:52.113751 8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b56fbef9a6438e8426df083e78d5d1" candidate_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1813e38e916647beb49a55a5bc392f01"
I20260809 07:53:52.113757 8254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b56fbef9a6438e8426df083e78d5d1" candidate_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f286b3371745e88d4efbcec859a619"
I20260809 07:53:52.113852 8254 raft_consensus.cc:3037] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:52.113857 8469 raft_consensus.cc:3037] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:52.114742 8254 raft_consensus.cc:2454] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ef4fcc7c19e4c7ca825f7930d1ae276 in term 1.
I20260809 07:53:52.114774 8469 raft_consensus.cc:2454] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ef4fcc7c19e4c7ca825f7930d1ae276 in term 1.
I20260809 07:53:52.115000 8577 leader_election.cc:302] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ef4fcc7c19e4c7ca825f7930d1ae276, a3f286b3371745e88d4efbcec859a619; no voters:
I20260809 07:53:52.115176 8789 raft_consensus.cc:2781] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:52.115264 8789 raft_consensus.cc:686] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 LEADER]: Becoming Leader. State: Replica: 5ef4fcc7c19e4c7ca825f7930d1ae276, State: Running, Role: LEADER
I20260809 07:53:52.115398 8789 consensus_queue.cc:237] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:52.116993 8024 catalog_manager.cc:5495] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ef4fcc7c19e4c7ca825f7930d1ae276 (127.28.112.3). New cstate: current_term: 1 leader_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } health_report { overall_health: HEALTHY } } }
I20260809 07:53:52.126677 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:52.123398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47040 (local address 127.28.112.3:39831)
0809 07:53:52.123512 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:52.123516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:52.123530 (+ 14us) server_negotiation.cc:407] Connection header received
0809 07:53:52.123660 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:52.123663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:52.123713 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:52.123771 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:52.124000 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.124504 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.125502 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.125782 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.125875 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:52.126248 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:52.126331 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:52.126397 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:52.126479 (+ 82us) server_negotiation.cc:299] Negotiation successful
0809 07:53:52.126534 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 07:53:52.128504 8469 raft_consensus.cc:1261] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01 [term 1 FOLLOWER]: Refusing update from remote peer 5ef4fcc7c19e4c7ca825f7930d1ae276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:52.128898 8254 raft_consensus.cc:1261] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619 [term 1 FOLLOWER]: Refusing update from remote peer 5ef4fcc7c19e4c7ca825f7930d1ae276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:52.128913 8789 consensus_queue.cc:1035] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:52.129281 8864 consensus_queue.cc:1035] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:52.135823 8023 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:52.135954 8023 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:52.137852 8023 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b: Sending DeleteTablet for 3 replicas of tablet a0b56fbef9a6438e8426df083e78d5d1
I20260809 07:53:52.138401 8440 tablet_service.cc:1514] Processing DeleteTablet for tablet a0b56fbef9a6438e8426df083e78d5d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:52 UTC) from {username='oryx', principal='oryx/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:59868
I20260809 07:53:52.138458 8641 tablet_service.cc:1514] Processing DeleteTablet for tablet a0b56fbef9a6438e8426df083e78d5d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:52 UTC) from {username='oryx', principal='oryx/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:47000
I20260809 07:53:52.138506 8223 tablet_service.cc:1514] Processing DeleteTablet for tablet a0b56fbef9a6438e8426df083e78d5d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:52 UTC) from {username='oryx', principal='oryx/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:50338
I20260809 07:53:52.138680 8897 tablet_replica.cc:330] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619: stopping tablet replica
I20260809 07:53:52.138696 8895 tablet_replica.cc:330] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01: stopping tablet replica
I20260809 07:53:52.138784 8897 raft_consensus.cc:2229] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:52.138808 8895 raft_consensus.cc:2229] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:52.138916 8897 raft_consensus.cc:2258] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:52.138929 8895 raft_consensus.cc:2258] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:52.139089 8894 tablet_replica.cc:330] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276: stopping tablet replica
I20260809 07:53:52.139210 8894 raft_consensus.cc:2229] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:53:52.139400 8894 raft_consensus.cc:2258] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:52.139595 8897 ts_tablet_manager.cc:1905] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:52.139864 8023 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 07:53:52.139997 8895 ts_tablet_manager.cc:1905] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 07:53:52.140199 8023 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:53:52.141137 8894 ts_tablet_manager.cc:1905] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:52.142237 8897 ts_tablet_manager.cc:1918] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:53:52.142311 8897 log.cc:1196] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-0/wal/wals/a0b56fbef9a6438e8426df083e78d5d1
I20260809 07:53:52.142405 8895 ts_tablet_manager.cc:1918] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:53:52.142465 8895 log.cc:1196] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/a0b56fbef9a6438e8426df083e78d5d1
I20260809 07:53:52.142558 8897 ts_tablet_manager.cc:1939] T a0b56fbef9a6438e8426df083e78d5d1 P a3f286b3371745e88d4efbcec859a619: Deleting consensus metadata
I20260809 07:53:52.142691 8895 ts_tablet_manager.cc:1939] T a0b56fbef9a6438e8426df083e78d5d1 P 1813e38e916647beb49a55a5bc392f01: Deleting consensus metadata
I20260809 07:53:52.143296 8005 catalog_manager.cc:4841] TS a3f286b3371745e88d4efbcec859a619 (127.28.112.1:42231): tablet a0b56fbef9a6438e8426df083e78d5d1 (table test-table [id=69afc1b6b8674de092cb074820c8bc99]) successfully deleted
I20260809 07:53:52.143345 8004 catalog_manager.cc:4841] TS 1813e38e916647beb49a55a5bc392f01 (127.28.112.2:45051): tablet a0b56fbef9a6438e8426df083e78d5d1 (table test-table [id=69afc1b6b8674de092cb074820c8bc99]) successfully deleted
I20260809 07:53:52.143627 8894 ts_tablet_manager.cc:1918] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:53:52.143695 8894 log.cc:1196] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/a0b56fbef9a6438e8426df083e78d5d1
I20260809 07:53:52.143880 8894 ts_tablet_manager.cc:1939] T a0b56fbef9a6438e8426df083e78d5d1 P 5ef4fcc7c19e4c7ca825f7930d1ae276: Deleting consensus metadata
I20260809 07:53:52.144450 8006 catalog_manager.cc:4841] TS 5ef4fcc7c19e4c7ca825f7930d1ae276 (127.28.112.3:39831): tablet a0b56fbef9a6438e8426df083e78d5d1 (table test-table [id=69afc1b6b8674de092cb074820c8bc99]) successfully deleted
I20260809 07:53:52.147156 8223 tablet_service.cc:1467] Processing CreateTablet for tablet de1099f0ca5c48a384c3ae0e19d3da3d (DEFAULT_TABLE table=test-table [id=26cc7ca4600e4164b67a0e720dd378b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:52.147224 8440 tablet_service.cc:1467] Processing CreateTablet for tablet de1099f0ca5c48a384c3ae0e19d3da3d (DEFAULT_TABLE table=test-table [id=26cc7ca4600e4164b67a0e720dd378b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:52.147409 8223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de1099f0ca5c48a384c3ae0e19d3da3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:52.147467 8440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de1099f0ca5c48a384c3ae0e19d3da3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:52.149190 8785 tablet_bootstrap.cc:492] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01: Bootstrap starting.
I20260809 07:53:52.149194 8783 tablet_bootstrap.cc:492] T de1099f0ca5c48a384c3ae0e19d3da3d P a3f286b3371745e88d4efbcec859a619: Bootstrap starting.
I20260809 07:53:52.149361 8641 tablet_service.cc:1467] Processing CreateTablet for tablet de1099f0ca5c48a384c3ae0e19d3da3d (DEFAULT_TABLE table=test-table [id=26cc7ca4600e4164b67a0e720dd378b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:53:52.149583 8641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de1099f0ca5c48a384c3ae0e19d3da3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:52.150013 8783 tablet_bootstrap.cc:654] T de1099f0ca5c48a384c3ae0e19d3da3d P a3f286b3371745e88d4efbcec859a619: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:52.150352 8785 tablet_bootstrap.cc:654] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:52.151050 8783 tablet_bootstrap.cc:492] T de1099f0ca5c48a384c3ae0e19d3da3d P a3f286b3371745e88d4efbcec859a619: No bootstrap required, opened a new log
I20260809 07:53:52.151130 8783 ts_tablet_manager.cc:1397] T de1099f0ca5c48a384c3ae0e19d3da3d P a3f286b3371745e88d4efbcec859a619: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:52.151458 8783 raft_consensus.cc:348] T de1099f0ca5c48a384c3ae0e19d3da3d P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:52.151546 8783 raft_consensus.cc:374] T de1099f0ca5c48a384c3ae0e19d3da3d P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:52.151575 8783 raft_consensus.cc:729] T de1099f0ca5c48a384c3ae0e19d3da3d P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f286b3371745e88d4efbcec859a619, State: Initialized, Role: FOLLOWER
I20260809 07:53:52.151693 8783 consensus_queue.cc:260] T de1099f0ca5c48a384c3ae0e19d3da3d P a3f286b3371745e88d4efbcec859a619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:52.151837 8782 tablet_bootstrap.cc:492] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: Bootstrap starting.
I20260809 07:53:52.151970 8783 ts_tablet_manager.cc:1428] T de1099f0ca5c48a384c3ae0e19d3da3d P a3f286b3371745e88d4efbcec859a619: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 07:53:52.152626 8782 tablet_bootstrap.cc:654] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:52.152786 8785 tablet_bootstrap.cc:492] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01: No bootstrap required, opened a new log
I20260809 07:53:52.152866 8785 ts_tablet_manager.cc:1397] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260809 07:53:52.153294 8785 raft_consensus.cc:348] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:52.153407 8785 raft_consensus.cc:374] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:52.153450 8785 raft_consensus.cc:729] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1813e38e916647beb49a55a5bc392f01, State: Initialized, Role: FOLLOWER
I20260809 07:53:52.153597 8785 consensus_queue.cc:260] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:52.153855 8785 ts_tablet_manager.cc:1428] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 07:53:52.154601 8782 tablet_bootstrap.cc:492] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: No bootstrap required, opened a new log
I20260809 07:53:52.154682 8782 ts_tablet_manager.cc:1397] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260809 07:53:52.155009 8782 raft_consensus.cc:348] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:52.155117 8782 raft_consensus.cc:374] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:52.155161 8782 raft_consensus.cc:729] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ef4fcc7c19e4c7ca825f7930d1ae276, State: Initialized, Role: FOLLOWER
I20260809 07:53:52.155300 8782 consensus_queue.cc:260] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:52.155562 8782 ts_tablet_manager.cc:1428] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 07:53:52.351257 8793 raft_consensus.cc:482] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:53:52.351401 8793 raft_consensus.cc:504] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:52.352553 8793 leader_election.cc:288] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3f286b3371745e88d4efbcec859a619 (127.28.112.1:42231), 5ef4fcc7c19e4c7ca825f7930d1ae276 (127.28.112.3:39831)
I20260809 07:53:52.356356 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:52.352914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:50749 (local address 127.28.112.3:39831)
0809 07:53:52.353042 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:52.353047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:52.353251 (+ 204us) server_negotiation.cc:407] Connection header received
0809 07:53:52.353365 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:52.353368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:52.353412 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:52.353488 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:52.353742 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.354281 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.355424 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.355799 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.355891 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:52.355951 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:52.356179 (+ 228us) server_negotiation.cc:299] Negotiation successful
0809 07:53:52.356246 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 07:53:52.356366 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:52.352825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:39831 (local address 127.28.112.2:50749)
0809 07:53:52.353157 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:52.353170 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:52.353281 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:52.353508 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:52.353511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:52.353528 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:52.353629 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:52.353633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.354404 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.354407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:52.355309 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:52.355318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.355932 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.355936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:52.356035 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:52.356060 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 07:53:52.356120 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 07:53:52.356187 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":71,"thread_start_us":63,"threads_started":1}
I20260809 07:53:52.356392 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:52.352871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:60135 (local address 127.28.112.1:42231)
0809 07:53:52.353160 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:52.353164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:52.353181 (+ 17us) server_negotiation.cc:407] Connection header received
0809 07:53:52.353292 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:52.353295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:52.353352 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:52.353428 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:53:52.353723 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.354279 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.355439 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.355845 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.355960 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:52.356015 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:52.356195 (+ 180us) server_negotiation.cc:299] Negotiation successful
0809 07:53:52.356278 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":78,"threads_started":1}
I20260809 07:53:52.356374 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:52.352767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:42231 (local address 127.28.112.2:60135)
0809 07:53:52.353072 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:52.353086 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:52.353178 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:52.353444 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:52.353447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:52.353464 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:53:52.353604 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:52.353611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.354383 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.354386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:52.355315 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:52.355325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.355960 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.355964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:52.356064 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:52.356082 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 07:53:52.356153 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 07:53:52.356221 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":37,"thread_start_us":78,"threads_started":1}
I20260809 07:53:52.357034 8674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1099f0ca5c48a384c3ae0e19d3da3d" candidate_uuid: "1813e38e916647beb49a55a5bc392f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" is_pre_election: true
I20260809 07:53:52.357034 8254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1099f0ca5c48a384c3ae0e19d3da3d" candidate_uuid: "1813e38e916647beb49a55a5bc392f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f286b3371745e88d4efbcec859a619" is_pre_election: true
I20260809 07:53:52.357188 8254 raft_consensus.cc:2454] T de1099f0ca5c48a384c3ae0e19d3da3d P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1813e38e916647beb49a55a5bc392f01 in term 0.
I20260809 07:53:52.357188 8674 raft_consensus.cc:2454] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1813e38e916647beb49a55a5bc392f01 in term 0.
I20260809 07:53:52.357563 8364 leader_election.cc:302] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1813e38e916647beb49a55a5bc392f01, 5ef4fcc7c19e4c7ca825f7930d1ae276; no voters:
I20260809 07:53:52.357821 8793 raft_consensus.cc:2781] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:53:52.357882 8793 raft_consensus.cc:482] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:53:52.357926 8793 raft_consensus.cc:3037] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:52.358897 8793 raft_consensus.cc:504] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:52.359227 8793 leader_election.cc:288] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [CANDIDATE]: Term 1 election: Requested vote from peers a3f286b3371745e88d4efbcec859a619 (127.28.112.1:42231), 5ef4fcc7c19e4c7ca825f7930d1ae276 (127.28.112.3:39831)
I20260809 07:53:52.359552 8674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1099f0ca5c48a384c3ae0e19d3da3d" candidate_uuid: "1813e38e916647beb49a55a5bc392f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276"
I20260809 07:53:52.359570 8254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1099f0ca5c48a384c3ae0e19d3da3d" candidate_uuid: "1813e38e916647beb49a55a5bc392f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f286b3371745e88d4efbcec859a619"
I20260809 07:53:52.359675 8254 raft_consensus.cc:3037] T de1099f0ca5c48a384c3ae0e19d3da3d P a3f286b3371745e88d4efbcec859a619 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:52.359673 8674 raft_consensus.cc:3037] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:52.360519 8254 raft_consensus.cc:2454] T de1099f0ca5c48a384c3ae0e19d3da3d P a3f286b3371745e88d4efbcec859a619 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1813e38e916647beb49a55a5bc392f01 in term 1.
I20260809 07:53:52.360673 8674 raft_consensus.cc:2454] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1813e38e916647beb49a55a5bc392f01 in term 1.
I20260809 07:53:52.360797 8363 leader_election.cc:302] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1813e38e916647beb49a55a5bc392f01, a3f286b3371745e88d4efbcec859a619; no voters:
I20260809 07:53:52.360973 8793 raft_consensus.cc:2781] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:52.361169 8793 raft_consensus.cc:686] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [term 1 LEADER]: Becoming Leader. State: Replica: 1813e38e916647beb49a55a5bc392f01, State: Running, Role: LEADER
I20260809 07:53:52.361420 8793 consensus_queue.cc:237] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } }
I20260809 07:53:52.363523 8024 catalog_manager.cc:5495] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1813e38e916647beb49a55a5bc392f01 (127.28.112.2). New cstate: current_term: 1 leader_uuid: "1813e38e916647beb49a55a5bc392f01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1813e38e916647beb49a55a5bc392f01" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 } health_report { overall_health: UNKNOWN } } }
I20260809 07:53:52.390661 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:52.386907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.28.112.2:45051)
0809 07:53:52.387177 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:52.387183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:52.387195 (+ 12us) server_negotiation.cc:407] Connection header received
0809 07:53:52.387262 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:52.387264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:52.387313 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:52.387414 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:52.387660 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.388232 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.389199 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.389465 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.389541 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:52.389746 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:52.390265 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:52.390352 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:52.390437 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 07:53:52.390494 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":130,"threads_started":1}
I20260809 07:53:52.400036 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:52.397339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:43309 (local address 127.28.112.3:39831)
0809 07:53:52.397451 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:52.397454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:52.397467 (+ 13us) server_negotiation.cc:407] Connection header received
0809 07:53:52.397574 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:52.397577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:52.397619 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:52.397677 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:52.397857 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.398354 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.399120 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.399331 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.399400 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:52.399712 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:52.399774 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:52.399817 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:52.399892 (+ 75us) server_negotiation.cc:299] Negotiation successful
0809 07:53:52.399932 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 07:53:52.400068 8947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:52.397153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:39831 (local address 127.28.112.2:43309)
0809 07:53:52.397401 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:52.397413 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:52.397492 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:52.397687 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:52.397689 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:52.397694 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 07:53:52.397789 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:52.397793 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.398426 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.398429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:52.399034 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:52.399042 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.399473 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.399477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:52.399548 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:52.399593 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 07:53:52.399828 (+ 235us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 07:53:52.399832 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 07:53:52.399876 (+ 44us) client_negotiation.cc:240] Negotiation successful
0809 07:53:52.399944 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":87,"threads_started":1}
I20260809 07:53:52.406401 8254 raft_consensus.cc:1261] T de1099f0ca5c48a384c3ae0e19d3da3d P a3f286b3371745e88d4efbcec859a619 [term 1 FOLLOWER]: Refusing update from remote peer 1813e38e916647beb49a55a5bc392f01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:52.406438 8674 raft_consensus.cc:1261] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 FOLLOWER]: Refusing update from remote peer 1813e38e916647beb49a55a5bc392f01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:53:52.406899 8793 consensus_queue.cc:1035] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3f286b3371745e88d4efbcec859a619" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 42231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:52.407105 8938 consensus_queue.cc:1035] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ef4fcc7c19e4c7ca825f7930d1ae276" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:53:52.427210 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:52.423832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:43091 (local address 127.28.112.2:45051)
0809 07:53:52.423940 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:52.423944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:52.423957 (+ 13us) server_negotiation.cc:407] Connection header received
0809 07:53:52.424002 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:52.424005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:52.424052 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:52.424126 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:53:52.424536 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.425132 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.426174 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.426394 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.426502 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:52.426826 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:53:52.426907 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:53:52.426965 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:52.427066 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 07:53:52.427111 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 07:53:52.427238 8954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:52.422199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:45051 (local address 127.28.112.3:43091)
0809 07:53:52.422410 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:52.422422 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:52.422497 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:52.424277 (+ 1780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:52.424281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:52.424290 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 07:53:52.424402 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:52.424409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.425256 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.425258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:52.426062 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:52.426073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:52.426589 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:52.426592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:52.426677 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:52.426711 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 07:53:52.426989 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 07:53:52.426998 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 07:53:52.427054 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 07:53:52.427127 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
I20260809 07:53:52.449707 8023 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:52.449858 8023 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:53:52.451876 8023 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3195bf3fdf00439f97686ff97571eb4b: Sending DeleteTablet for 3 replicas of tablet de1099f0ca5c48a384c3ae0e19d3da3d
I20260809 07:53:52.452488 8223 tablet_service.cc:1514] Processing DeleteTablet for tablet de1099f0ca5c48a384c3ae0e19d3da3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:52 UTC) from {username='oryx', principal='oryx/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:50338
I20260809 07:53:52.452824 8641 tablet_service.cc:1514] Processing DeleteTablet for tablet de1099f0ca5c48a384c3ae0e19d3da3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:52 UTC) from {username='oryx', principal='oryx/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:47000
I20260809 07:53:52.453035 8894 tablet_replica.cc:330] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: stopping tablet replica
I20260809 07:53:52.453163 8894 raft_consensus.cc:2229] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:53:52.453184 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 8128
I20260809 07:53:52.453279 8894 raft_consensus.cc:2258] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:52.459977 8440 tablet_service.cc:1514] Processing DeleteTablet for tablet de1099f0ca5c48a384c3ae0e19d3da3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:53:52 UTC) from {username='oryx', principal='oryx/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:59868
I20260809 07:53:52.460148 8895 tablet_replica.cc:330] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01: stopping tablet replica
I20260809 07:53:52.460238 8895 raft_consensus.cc:2229] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:53:52.460531 8895 raft_consensus.cc:2258] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:53:52.461517 8894 ts_tablet_manager.cc:1905] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:52.464543 8894 ts_tablet_manager.cc:1918] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 07:53:52.464660 8894 log.cc:1196] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/de1099f0ca5c48a384c3ae0e19d3da3d
I20260809 07:53:52.464964 8894 ts_tablet_manager.cc:1939] T de1099f0ca5c48a384c3ae0e19d3da3d P 5ef4fcc7c19e4c7ca825f7930d1ae276: Deleting consensus metadata
I20260809 07:53:52.465874 8006 catalog_manager.cc:4841] TS 5ef4fcc7c19e4c7ca825f7930d1ae276 (127.28.112.3:39831): tablet de1099f0ca5c48a384c3ae0e19d3da3d (table test-table [id=26cc7ca4600e4164b67a0e720dd378b2]) successfully deleted
I20260809 07:53:52.466498 8895 ts_tablet_manager.cc:1905] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:53:52.468856 8895 ts_tablet_manager.cc:1918] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 07:53:52.468938 8895 log.cc:1196] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/de1099f0ca5c48a384c3ae0e19d3da3d
I20260809 07:53:52.469170 8895 ts_tablet_manager.cc:1939] T de1099f0ca5c48a384c3ae0e19d3da3d P 1813e38e916647beb49a55a5bc392f01: Deleting consensus metadata
I20260809 07:53:52.469872 8004 catalog_manager.cc:4841] TS 1813e38e916647beb49a55a5bc392f01 (127.28.112.2:45051): tablet de1099f0ca5c48a384c3ae0e19d3da3d (table test-table [id=26cc7ca4600e4164b67a0e720dd378b2]) successfully deleted
W20260809 07:53:52.473403 8005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.112.1:42231 (error 108)
I20260809 07:53:52.473836 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 8334
I20260809 07:53:52.478849 8958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:52.476509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:42231 (local address 127.0.0.1:50354)
0809 07:53:52.478692 (+ 2183us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:52.478762 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2102,"thread_start_us":749,"threads_started":1}
W20260809 07:53:52.479089 8005 catalog_manager.cc:4584] TS a3f286b3371745e88d4efbcec859a619 (127.28.112.1:42231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de1099f0ca5c48a384c3ae0e19d3da3d: Network error: Client connection negotiation failed: client connection to 127.28.112.1:42231: connect: Connection refused (error 111)
I20260809 07:53:52.484356 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 8550
I20260809 07:53:52.501109 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 7973
2026-08-09T07:53:52Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3803 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:53:52 dist-test-slave-vpvm krb5kdc[8964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:53:52 dist-test-slave-vpvm krb5kdc[8964](info): set up 2 sockets
Aug 09 07:53:52 dist-test-slave-vpvm krb5kdc[8964](info): commencing operation
krb5kdc: starting...
W20260809 07:53:53.654531 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:53:53 dist-test-slave-vpvm krb5kdc[8964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:53:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:53:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:53:53.768541 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:37391
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:37391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:53.877439 8984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:53.877669 8984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:53.877722 8984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:53.881580 8984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:53.881633 8984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:53.881654 8984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:53.881675 8984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:53.881708 8984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:53:53.886781 8984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:37391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:37391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:53.887769 8984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:53.888896 8984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:53.896845 8989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:53.896849 8990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:53.896852 8994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:53.897068 8984 server_base.cc:1034] running on GCE node
I20260809 07:53:53.897603 8984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:53.898628 8984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:53.899796 8984 hybrid_clock.cc:648] HybridClock initialized: now 1786262033899764 us; error 40 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-09T07:53:53Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1408 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:53:53 dist-test-slave-vpvm krb5kdc[8999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:53:53 dist-test-slave-vpvm krb5kdc[8999](info): set up 2 sockets
Aug 09 07:53:53 dist-test-slave-vpvm krb5kdc[8999](info): commencing operation
krb5kdc: starting...
W20260809 07:53:55.068117 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:53:55 dist-test-slave-vpvm krb5kdc[8999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:53:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:53:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:53:55.182799 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:38715
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:38715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:55.291185 9021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:55.291425 9021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:55.291464 9021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:55.295356 9021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:55.295403 9021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:55.295421 9021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:55.295436 9021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:55.295451 9021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:53:55.300539 9021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:38715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:38715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:55.301486 9021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:55.302493 9021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:55.310564 9032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:55.310611 9026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:55.310738 9021 server_base.cc:1034] running on GCE node
W20260809 07:53:55.310578 9030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:55.311350 9021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:55.312351 9021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:55.313580 9021 hybrid_clock.cc:648] HybridClock initialized: now 1786262035313547 us; error 38 us; skew 500 ppm
Aug 09 07:53:55 dist-test-slave-vpvm krb5kdc[8999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:55.316277 9021 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:53:55.317111 9021 webserver.cc:460] Webserver started at http://127.28.112.62:45359/ using document root <none> and password file <none>
I20260809 07:53:55.317612 9021 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:55.317668 9021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:55.317830 9021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:55.319821 9021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "ced474ac584e4d1e84e3945db20cbc4f"
format_stamp: "Formatted at 2026-08-09 07:53:55 on dist-test-slave-vpvm"
server_key: "36ce939457d329c48112da3560c6c6b3"
server_key_iv: "ebfa459390114d573bbf8863df5aa98b"
server_key_version: "encryptionkey@0"
I20260809 07:53:55.320319 9021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "ced474ac584e4d1e84e3945db20cbc4f"
format_stamp: "Formatted at 2026-08-09 07:53:55 on dist-test-slave-vpvm"
server_key: "36ce939457d329c48112da3560c6c6b3"
server_key_iv: "ebfa459390114d573bbf8863df5aa98b"
server_key_version: "encryptionkey@0"
I20260809 07:53:55.323891 9021 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 07:53:55.326059 9042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:55.327195 9021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:55.327293 9021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "ced474ac584e4d1e84e3945db20cbc4f"
format_stamp: "Formatted at 2026-08-09 07:53:55 on dist-test-slave-vpvm"
server_key: "36ce939457d329c48112da3560c6c6b3"
server_key_iv: "ebfa459390114d573bbf8863df5aa98b"
server_key_version: "encryptionkey@0"
I20260809 07:53:55.327371 9021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:55.341236 9021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:55.343712 9021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:55.343851 9021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:55.350358 9021 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:38715
I20260809 07:53:55.350409 9112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:38715 every 8 connection(s)
I20260809 07:53:55.351238 9021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:53:55.353899 9118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:53:55.358233 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 9021
I20260809 07:53:55.358369 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:53:55.358680 29120 external_mini_cluster.cc:1428] Setting key 1ce4b9be7df903eeab38f01f4aecec99
I20260809 07:53:55.359581 9118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f: Bootstrap starting.
I20260809 07:53:55.362229 9118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f: Neither blocks nor log segments found. Creating new log.
I20260809 07:53:55.363049 9118 log.cc:824] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:55.364918 9118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f: No bootstrap required, opened a new log
Aug 09 07:53:55 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:55.368608 9118 raft_consensus.cc:348] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced474ac584e4d1e84e3945db20cbc4f" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38715 } }
I20260809 07:53:55.368892 9118 raft_consensus.cc:374] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:53:55.368953 9118 raft_consensus.cc:729] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced474ac584e4d1e84e3945db20cbc4f, State: Initialized, Role: FOLLOWER
I20260809 07:53:55.369460 9118 consensus_queue.cc:260] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced474ac584e4d1e84e3945db20cbc4f" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38715 } }
I20260809 07:53:55.369621 9118 raft_consensus.cc:388] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:53:55.369701 9118 raft_consensus.cc:482] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:53:55.369799 9118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:53:55.370769 9118 raft_consensus.cc:504] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced474ac584e4d1e84e3945db20cbc4f" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38715 } }
I20260809 07:53:55.371119 9118 leader_election.cc:302] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ced474ac584e4d1e84e3945db20cbc4f; no voters:
I20260809 07:53:55.371445 9118 leader_election.cc:288] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:53:55.371531 9125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:53:55.371780 9125 raft_consensus.cc:686] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [term 1 LEADER]: Becoming Leader. State: Replica: ced474ac584e4d1e84e3945db20cbc4f, State: Running, Role: LEADER
I20260809 07:53:55.372170 9125 consensus_queue.cc:237] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced474ac584e4d1e84e3945db20cbc4f" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38715 } }
I20260809 07:53:55.372782 9118 sys_catalog.cc:564] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:55.373076 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:55.359999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34924 (local address 127.28.112.62:38715)
0809 07:53:55.360400 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:55.360417 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:55.360457 (+ 40us) server_negotiation.cc:407] Connection header received
0809 07:53:55.361257 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:55.361281 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:55.361666 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:55.362036 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:55.362740 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:55.364045 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:55.364915 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:55.365585 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:55.365747 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:55.368053 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:55.368059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:55.368075 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:55.368106 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:55.370032 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:55.370439 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:55.370445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:55.370452 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:55.370601 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:55.370829 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:55.370833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:55.370834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:55.371361 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:55.371563 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:55.371886 (+ 323us) server_negotiation.cc:299] Negotiation successful
0809 07:53:55.372176 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":104,"threads_started":1}
I20260809 07:53:55.373499 9126 sys_catalog.cc:455] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ced474ac584e4d1e84e3945db20cbc4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced474ac584e4d1e84e3945db20cbc4f" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38715 } } }
I20260809 07:53:55.374280 9126 sys_catalog.cc:458] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:55.373227 9125 sys_catalog.cc:455] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ced474ac584e4d1e84e3945db20cbc4f. Latest consensus state: current_term: 1 leader_uuid: "ced474ac584e4d1e84e3945db20cbc4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced474ac584e4d1e84e3945db20cbc4f" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38715 } } }
I20260809 07:53:55.375413 9125 sys_catalog.cc:458] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [sys.catalog]: This master's current role is: LEADER
W20260809 07:53:55.377393 9143 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:53:55.377467 9143 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:53:55.377533 9139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:53:55.378269 9139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:55.383319 9139 catalog_manager.cc:1347] Generated new cluster ID: e2104f8bcfa049239f57ca1c0da7bab8
I20260809 07:53:55.383378 9139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:55.398648 9139 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:53:55.399799 9139 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:55.407783 9139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f: Generated new TSK 0
I20260809 07:53:55.408506 9139 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
I20260809 07:53:55.442128 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:38715
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:55.550866 9152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:55.551095 9152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:55.551138 9152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:55.555018 9152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:55.555091 9152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:55.555173 9152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:53:55.560245 9152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:38715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:55.561183 9152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:55.562175 9152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:55.571184 9164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:55.571177 9163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:55.571285 9152 server_base.cc:1034] running on GCE node
W20260809 07:53:55.571177 9167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:55.571897 9152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:55.572506 9152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:55.573686 9152 hybrid_clock.cc:648] HybridClock initialized: now 1786262035573671 us; error 27 us; skew 500 ppm
Aug 09 07:53:55 dist-test-slave-vpvm krb5kdc[8999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:55.576366 9152 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:53:55.577157 9152 webserver.cc:460] Webserver started at http://127.28.112.1:37917/ using document root <none> and password file <none>
I20260809 07:53:55.577651 9152 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:55.577706 9152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:55.577867 9152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:55.579782 9152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "9171be83788845aa9db9941ee4487bbb"
format_stamp: "Formatted at 2026-08-09 07:53:55 on dist-test-slave-vpvm"
server_key: "087f94423d416b4483b287c46e1b993a"
server_key_iv: "dbfefe8f3841679c507da4d626b6a9b6"
server_key_version: "encryptionkey@0"
I20260809 07:53:55.580276 9152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "9171be83788845aa9db9941ee4487bbb"
format_stamp: "Formatted at 2026-08-09 07:53:55 on dist-test-slave-vpvm"
server_key: "087f94423d416b4483b287c46e1b993a"
server_key_iv: "dbfefe8f3841679c507da4d626b6a9b6"
server_key_version: "encryptionkey@0"
I20260809 07:53:55.583750 9152 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 07:53:55.585992 9177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:55.587111 9152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:55.587211 9152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "9171be83788845aa9db9941ee4487bbb"
format_stamp: "Formatted at 2026-08-09 07:53:55 on dist-test-slave-vpvm"
server_key: "087f94423d416b4483b287c46e1b993a"
server_key_iv: "dbfefe8f3841679c507da4d626b6a9b6"
server_key_version: "encryptionkey@0"
I20260809 07:53:55.587301 9152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:55.600755 9152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:55.603242 9152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:55.603402 9152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:55.603941 9152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:55.604732 9152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:55.604779 9152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:55.604821 9152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:55.604851 9152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:55.612984 9152 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:39951
I20260809 07:53:55.613008 9343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:39951 every 8 connection(s)
I20260809 07:53:55.613806 9152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:53:55.617712 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 9152
I20260809 07:53:55.617851 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:53:55.618204 29120 external_mini_cluster.cc:1428] Setting key 2255be68176b416ea998adee4431b310
Aug 09 07:53:55 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:55.627799 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:55.615590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:51045 (local address 127.28.112.62:38715)
0809 07:53:55.615726 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:55.615730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:55.616531 (+ 801us) server_negotiation.cc:407] Connection header received
0809 07:53:55.617889 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:55.617893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:55.617979 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:55.618068 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:55.619373 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:55.619896 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:55.620995 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:55.621224 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:55.621323 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:55.623974 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:55.623977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:55.623979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:55.624005 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:55.625410 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:55.626061 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:55.626064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:55.626066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:55.626151 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:55.626606 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:55.626610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:55.626611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:55.626809 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:55.626902 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:55.627557 (+ 655us) server_negotiation.cc:299] Negotiation successful
0809 07:53:55.627673 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.28.112.2@KRBTEST.COM; defaulting to no policy
I20260809 07:53:55.628837 9347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:55.615936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.1:51045)
0809 07:53:55.616380 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:55.616420 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:55.617631 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:55.618298 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:55.618307 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:55.618734 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:55.619224 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:55.619238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:55.620015 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:55.620018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:55.620877 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:55.620886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:55.621382 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:55.621388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:55.621590 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:55.622155 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:55.622172 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:55.623783 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:55.625537 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:55.625543 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:55.625559 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:55.625921 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:55.626246 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:55.626249 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:55.626252 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:55.626440 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:55.626927 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:55.626935 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:55.627253 (+ 318us) client_negotiation.cc:769] Sending connection context
0809 07:53:55.627489 (+ 236us) client_negotiation.cc:240] Negotiation successful
0809 07:53:55.627862 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":119,"threads_started":1}
I20260809 07:53:55.630062 9344 heartbeater.cc:347] Connected to a master server at 127.28.112.62:38715
I20260809 07:53:55.630280 9344 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:55.630760 9344 heartbeater.cc:511] Master 127.28.112.62:38715 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.2@KRBTEST.COM" created.
I20260809 07:53:55.632402 9066 ts_manager.cc:194] Registered new tserver with Master: 9171be83788845aa9db9941ee4487bbb (127.28.112.1:39951)
I20260809 07:53:55.633690 9066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:51045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
I20260809 07:53:55.645027 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:38715
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:55.751060 9350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:55.751271 9350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:55.751308 9350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:55.755088 9350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:55.755128 9350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:55.755180 9350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:53:55.760041 9350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:38715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:55.760860 9350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:55.761723 9350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:55.770320 9361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:55.770381 9350 server_base.cc:1034] running on GCE node
W20260809 07:53:55.770321 9359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:55.770315 9358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:55.770907 9350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:55.771494 9350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:55.772646 9350 hybrid_clock.cc:648] HybridClock initialized: now 1786262035772630 us; error 28 us; skew 500 ppm
Aug 09 07:53:55 dist-test-slave-vpvm krb5kdc[8999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:55.775292 9350 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:53:55.776059 9350 webserver.cc:460] Webserver started at http://127.28.112.2:36339/ using document root <none> and password file <none>
I20260809 07:53:55.776547 9350 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:55.776612 9350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:55.776816 9350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:55.778756 9350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "a36d5f1d7f3b4236993d68b6c06d05d4"
format_stamp: "Formatted at 2026-08-09 07:53:55 on dist-test-slave-vpvm"
server_key: "e47b140e13f00cd9c271715a51b93f2f"
server_key_iv: "1abc3c9e8ee72080bd9c6e47d4c3d4c4"
server_key_version: "encryptionkey@0"
I20260809 07:53:55.779259 9350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "a36d5f1d7f3b4236993d68b6c06d05d4"
format_stamp: "Formatted at 2026-08-09 07:53:55 on dist-test-slave-vpvm"
server_key: "e47b140e13f00cd9c271715a51b93f2f"
server_key_iv: "1abc3c9e8ee72080bd9c6e47d4c3d4c4"
server_key_version: "encryptionkey@0"
I20260809 07:53:55.782816 9350 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:53:55.785229 9368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:55.786525 9350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:55.786623 9350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "a36d5f1d7f3b4236993d68b6c06d05d4"
format_stamp: "Formatted at 2026-08-09 07:53:55 on dist-test-slave-vpvm"
server_key: "e47b140e13f00cd9c271715a51b93f2f"
server_key_iv: "1abc3c9e8ee72080bd9c6e47d4c3d4c4"
server_key_version: "encryptionkey@0"
I20260809 07:53:55.786734 9350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:55.809928 9350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:55.812566 9350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:55.812732 9350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:55.813290 9350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:55.814126 9350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:55.814172 9350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:55.814216 9350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:55.814245 9350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:55.822456 9350 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:38243
I20260809 07:53:55.822482 9537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:38243 every 8 connection(s)
I20260809 07:53:55.823289 9350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:53:55.830650 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 9350
I20260809 07:53:55.830787 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:53:55.831136 29120 external_mini_cluster.cc:1428] Setting key ce513e2439da26f3e85b5b707b931505
Aug 09 07:53:55 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:55.836715 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:55.825122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:44951 (local address 127.28.112.62:38715)
0809 07:53:55.825264 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:55.825268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:55.826113 (+ 845us) server_negotiation.cc:407] Connection header received
0809 07:53:55.827388 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:55.827391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:55.827437 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:55.827499 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:55.828525 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:55.829029 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:55.830127 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:55.830332 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:55.830419 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:55.833105 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:55.833109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:55.833112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:55.833145 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:55.834613 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:55.835296 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:55.835299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:55.835300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:55.835375 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:55.835704 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:55.835705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:55.835707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:55.835865 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:55.835937 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:55.836426 (+ 489us) server_negotiation.cc:299] Negotiation successful
0809 07:53:55.836544 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 07:53:55.837378 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:55.825485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.2:44951)
0809 07:53:55.825938 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:55.826016 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:55.827192 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:55.827641 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:55.827647 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:55.827975 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:55.828409 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:55.828421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:55.829114 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:55.829116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:55.830034 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:55.830053 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:55.830599 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:55.830604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:55.830822 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:55.831375 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:55.831396 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:55.832950 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:55.834861 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:55.834868 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:55.834882 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:55.835207 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:55.835454 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:55.835457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:55.835459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:55.835627 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:55.835958 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:55.835965 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:55.836198 (+ 233us) client_negotiation.cc:769] Sending connection context
0809 07:53:55.836347 (+ 149us) client_negotiation.cc:240] Negotiation successful
0809 07:53:55.836575 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":132,"threads_started":1}
I20260809 07:53:55.838610 9538 heartbeater.cc:347] Connected to a master server at 127.28.112.62:38715
I20260809 07:53:55.838840 9538 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:55.839279 9538 heartbeater.cc:511] Master 127.28.112.62:38715 requested a full tablet report, sending...
I20260809 07:53:55.840306 9066 ts_manager.cc:194] Registered new tserver with Master: a36d5f1d7f3b4236993d68b6c06d05d4 (127.28.112.2:38243)
I20260809 07:53:55.840843 9066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:44951
WARNING: no policy specified for kudu/127.28.112.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
I20260809 07:53:55.857924 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:38715
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:55.970563 9552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:55.970815 9552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:55.970883 9552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:55.974903 9552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:55.974964 9552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:55.975044 9552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:53:55.980116 9552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:38715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:55.981106 9552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:55.982120 9552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:55.991526 9563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:55.991532 9566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:55.991696 9552 server_base.cc:1034] running on GCE node
W20260809 07:53:55.991526 9561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:55.992180 9552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:55.992802 9552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:55.993999 9552 hybrid_clock.cc:648] HybridClock initialized: now 1786262035993983 us; error 28 us; skew 500 ppm
Aug 09 07:53:55 dist-test-slave-vpvm krb5kdc[8999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:55.996644 9552 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:53:55.997501 9552 webserver.cc:460] Webserver started at http://127.28.112.3:37317/ using document root <none> and password file <none>
I20260809 07:53:55.998075 9552 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:55.998147 9552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:55.998346 9552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:53:56.000289 9552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "d85aca7b375f4863b80e2d7b7889c062"
format_stamp: "Formatted at 2026-08-09 07:53:55 on dist-test-slave-vpvm"
server_key: "e796760b4d115116530090912f91d288"
server_key_iv: "8abc2f6170f2292315bc3ec1b7e7f9e1"
server_key_version: "encryptionkey@0"
I20260809 07:53:56.000806 9552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "d85aca7b375f4863b80e2d7b7889c062"
format_stamp: "Formatted at 2026-08-09 07:53:55 on dist-test-slave-vpvm"
server_key: "e796760b4d115116530090912f91d288"
server_key_iv: "8abc2f6170f2292315bc3ec1b7e7f9e1"
server_key_version: "encryptionkey@0"
I20260809 07:53:56.004415 9552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 07:53:56.006763 9576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:56.007887 9552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:56.008004 9552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "d85aca7b375f4863b80e2d7b7889c062"
format_stamp: "Formatted at 2026-08-09 07:53:55 on dist-test-slave-vpvm"
server_key: "e796760b4d115116530090912f91d288"
server_key_iv: "8abc2f6170f2292315bc3ec1b7e7f9e1"
server_key_version: "encryptionkey@0"
I20260809 07:53:56.008092 9552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:56.018452 9552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:56.021278 9552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:56.021448 9552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:56.021996 9552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:56.022750 9552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:56.022796 9552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:56.022845 9552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:56.022876 9552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:56.031270 9552 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:38743
I20260809 07:53:56.031322 9777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:38743 every 8 connection(s)
I20260809 07:53:56.032125 9552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:53:56.034977 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 9552
I20260809 07:53:56.035122 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:53:56.035444 29120 external_mini_cluster.cc:1428] Setting key cdbc5c21673b7b3c792ababb05bbf8a2
Aug 09 07:53:56 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:53:56.044484 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:56.033749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:45809 (local address 127.28.112.62:38715)
0809 07:53:56.033904 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:56.033923 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:56.034712 (+ 789us) server_negotiation.cc:407] Connection header received
0809 07:53:56.036213 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:56.036217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:56.036273 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:56.036346 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:56.037510 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.038080 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.039124 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.039320 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.039387 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:56.041514 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:56.041517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:56.041519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:56.041544 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:56.042925 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:56.043343 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:56.043345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:56.043346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:56.043415 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:56.043660 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:56.043662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:56.043663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:56.043817 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:56.043883 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:56.044273 (+ 390us) server_negotiation.cc:299] Negotiation successful
0809 07:53:56.044363 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 07:53:56.045189 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:56.034135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.3:45809)
0809 07:53:56.034562 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:56.034601 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:56.036039 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:56.036497 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:56.036504 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:56.036891 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:56.037368 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:56.037387 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.038177 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.038180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:56.039036 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:56.039046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.039420 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.039422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:56.039556 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:56.040023 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:56.040039 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:56.041370 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:53:56.043006 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:56.043010 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:56.043020 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:56.043271 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:56.043474 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:53:56.043476 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:53:56.043478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:53:56.043595 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:53:56.043883 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:53:56.043887 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:53:56.044097 (+ 210us) client_negotiation.cc:769] Sending connection context
0809 07:53:56.044239 (+ 142us) client_negotiation.cc:240] Negotiation successful
0809 07:53:56.044460 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":107,"threads_started":1}
I20260809 07:53:56.046240 9780 heartbeater.cc:347] Connected to a master server at 127.28.112.62:38715
I20260809 07:53:56.046437 9780 heartbeater.cc:464] Registering TS with master...
I20260809 07:53:56.046855 9780 heartbeater.cc:511] Master 127.28.112.62:38715 requested a full tablet report, sending...
I20260809 07:53:56.047814 9066 ts_manager.cc:194] Registered new tserver with Master: d85aca7b375f4863b80e2d7b7889c062 (127.28.112.3:38743)
I20260809 07:53:56.048486 9066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:45809
I20260809 07:53:56.049126 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.1.keytab.
Entry for principal oryx/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.1.keytab.
WARNING: no policy specified for oryx/127.28.112.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.112.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.2.keytab.
Entry for principal oryx/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.2.keytab.
WARNING: no policy specified for oryx/127.28.112.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.112.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.3.keytab.
Entry for principal oryx/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.3.keytab.
I20260809 07:53:56.114741 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 9152
I20260809 07:53:56.127147 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 9350
I20260809 07:53:56.139559 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 9552
I20260809 07:53:56.149297 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 9021
I20260809 07:53:56.158398 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:38715
--webserver_interface=127.28.112.62
--webserver_port=45359
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:38715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:53:56.267946 9791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:56.268200 9791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:56.268258 9791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:56.272173 9791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:53:56.272230 9791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:56.272257 9791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:56.272279 9791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:53:56.272305 9791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:53:56.277413 9791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:38715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:38715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=45359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:56.278420 9791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:56.279397 9791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:56.287616 9799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:56.287640 9791 server_base.cc:1034] running on GCE node
W20260809 07:53:56.287592 9798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:56.287603 9802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:56.288359 9791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:56.289381 9791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:56.290534 9791 hybrid_clock.cc:648] HybridClock initialized: now 1786262036290508 us; error 37 us; skew 500 ppm
Aug 09 07:53:56 dist-test-slave-vpvm krb5kdc[8999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262036, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:56.293087 9791 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:53:56.293918 9791 webserver.cc:460] Webserver started at http://127.28.112.62:45359/ using document root <none> and password file <none>
I20260809 07:53:56.294445 9791 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:56.294518 9791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:56.298143 9791 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260809 07:53:56.300129 9811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:56.301257 9791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:56.301373 9791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "ced474ac584e4d1e84e3945db20cbc4f"
format_stamp: "Formatted at 2026-08-09 07:53:55 on dist-test-slave-vpvm"
server_key: "36ce939457d329c48112da3560c6c6b3"
server_key_iv: "ebfa459390114d573bbf8863df5aa98b"
server_key_version: "encryptionkey@0"
I20260809 07:53:56.301864 9791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:56.318749 9791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:56.321393 9791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:56.321569 9791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:56.328424 9791 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:38715
I20260809 07:53:56.328440 9896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:38715 every 8 connection(s)
I20260809 07:53:56.329492 9791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:53:56.333716 9898 sys_catalog.cc:263] Verifying existing consensus state
I20260809 07:53:56.334178 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 9791
I20260809 07:53:56.334667 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:39951
--local_ip_for_outbound_sockets=127.28.112.1
--tserver_master_addrs=127.28.112.62:38715
--webserver_port=37917
--webserver_interface=127.28.112.1
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.112.1
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 07:53:56.335783 9898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f: Bootstrap starting.
I20260809 07:53:56.351704 9898 log.cc:824] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f: Log is configured to *not* fsync() on all Append() calls
I20260809 07:53:56.358101 9898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 07:53:56.358414 9898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f: Bootstrap complete.
I20260809 07:53:56.362346 9898 raft_consensus.cc:348] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced474ac584e4d1e84e3945db20cbc4f" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38715 } }
I20260809 07:53:56.362731 9898 raft_consensus.cc:729] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced474ac584e4d1e84e3945db20cbc4f, State: Initialized, Role: FOLLOWER
I20260809 07:53:56.363204 9898 consensus_queue.cc:260] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced474ac584e4d1e84e3945db20cbc4f" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38715 } }
I20260809 07:53:56.363313 9898 raft_consensus.cc:388] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:53:56.363371 9898 raft_consensus.cc:482] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:53:56.363469 9898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [term 1 FOLLOWER]: Advancing to term 2
I20260809 07:53:56.365231 9898 raft_consensus.cc:504] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced474ac584e4d1e84e3945db20cbc4f" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38715 } }
I20260809 07:53:56.365546 9898 leader_election.cc:302] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ced474ac584e4d1e84e3945db20cbc4f; no voters:
I20260809 07:53:56.365820 9898 leader_election.cc:288] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 07:53:56.365919 9902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 07:53:56.366137 9902 raft_consensus.cc:686] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [term 2 LEADER]: Becoming Leader. State: Replica: ced474ac584e4d1e84e3945db20cbc4f, State: Running, Role: LEADER
I20260809 07:53:56.366425 9902 consensus_queue.cc:237] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced474ac584e4d1e84e3945db20cbc4f" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38715 } }
I20260809 07:53:56.367081 9898 sys_catalog.cc:564] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:53:56.367357 9902 sys_catalog.cc:455] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ced474ac584e4d1e84e3945db20cbc4f. Latest consensus state: current_term: 2 leader_uuid: "ced474ac584e4d1e84e3945db20cbc4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced474ac584e4d1e84e3945db20cbc4f" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38715 } } }
I20260809 07:53:56.367470 9902 sys_catalog.cc:458] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:56.367867 9903 sys_catalog.cc:455] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ced474ac584e4d1e84e3945db20cbc4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced474ac584e4d1e84e3945db20cbc4f" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38715 } } }
I20260809 07:53:56.368060 9903 sys_catalog.cc:458] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f [sys.catalog]: This master's current role is: LEADER
I20260809 07:53:56.368391 9907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:53:56.372080 9907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:53:56.372694 9907 catalog_manager.cc:1259] Loaded cluster ID: e2104f8bcfa049239f57ca1c0da7bab8
I20260809 07:53:56.372754 9907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:53:56.375272 9907 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:53:56.375967 9907 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ced474ac584e4d1e84e3945db20cbc4f: Loaded TSK: 0
I20260809 07:53:56.377185 9907 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 07:53:56.445073 9900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:56.445298 9900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:56.445355 9900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:56.449232 9900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:56.449283 9900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:56.449355 9900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:53:56.454276 9900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.1.keytab
--principal=oryx/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:39951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=37917
--tserver_master_addrs=127.28.112.62:38715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:56.455137 9900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:56.456032 9900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:56.464785 9938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:56.464802 9939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:56.465233 9942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:56.465315 9900 server_base.cc:1034] running on GCE node
I20260809 07:53:56.465708 9900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:56.466342 9900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:56.467515 9900 hybrid_clock.cc:648] HybridClock initialized: now 1786262036467491 us; error 33 us; skew 500 ppm
Aug 09 07:53:56 dist-test-slave-vpvm krb5kdc[8999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262036, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:56.470317 9900 init.cc:377] Logged in from keytab as oryx/127.28.112.1@KRBTEST.COM (short username oryx)
I20260809 07:53:56.471205 9900 webserver.cc:460] Webserver started at http://127.28.112.1:37917/ using document root <none> and password file <none>
I20260809 07:53:56.471721 9900 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:56.471792 9900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:56.475474 9900 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:56.477442 9952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:56.478634 9900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:53:56.478725 9900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "9171be83788845aa9db9941ee4487bbb"
format_stamp: "Formatted at 2026-08-09 07:53:55 on dist-test-slave-vpvm"
server_key: "087f94423d416b4483b287c46e1b993a"
server_key_iv: "dbfefe8f3841679c507da4d626b6a9b6"
server_key_version: "encryptionkey@0"
I20260809 07:53:56.479239 9900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:56.496198 9900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:56.499260 9900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:56.499428 9900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:56.500026 9900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:56.500890 9900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:56.500939 9900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:56.500982 9900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:56.501020 9900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:56.509559 9900 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:39951
I20260809 07:53:56.509591 10114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:39951 every 8 connection(s)
I20260809 07:53:56.510524 9900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:53:56.511420 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 9900
I20260809 07:53:56.511940 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:38243
--local_ip_for_outbound_sockets=127.28.112.2
--tserver_master_addrs=127.28.112.62:38715
--webserver_port=36339
--webserver_interface=127.28.112.2
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.112.2
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 07:53:56 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:53:56 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 07:53:56.524700 10119 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:56.512728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:33587 (local address 127.28.112.62:38715)
0809 07:53:56.513219 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:56.513239 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:56.513824 (+ 585us) server_negotiation.cc:407] Connection header received
0809 07:53:56.514898 (+ 1074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:56.514908 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:56.515189 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:56.515472 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:56.516820 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.517826 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.519489 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.519960 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.520090 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:56.524028 (+ 3938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.1:33587: BlockingRecv error: recv got EOF from 127.28.112.1:33587 (error 108)
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":126,"threads_started":1}
I20260809 07:53:56.524798 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:56.512806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.1:33587)
0809 07:53:56.513609 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:56.513703 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:56.514526 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:56.515580 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:56.515588 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:56.516035 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:56.516636 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:56.516655 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.517999 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.518003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:56.519338 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:56.519351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.520129 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.520134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:56.520326 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:56.520972 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:56.520993 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:56.523812 (+ 2819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":138,"threads_started":1}
W20260809 07:53:56.525240 10115 heartbeater.cc:644] Failed to heartbeat to 127.28.112.62:38715 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.112.62:38715: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Aug 09 07:53:56 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:53:56 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:53:56.530687 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:56.525829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.1:43345)
0809 07:53:56.525931 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:56.525945 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:56.526042 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:56.526250 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:56.526252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:56.526409 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:56.526526 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:56.526532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.527184 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.527187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:56.528064 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:56.528073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.528454 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.528457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:56.528554 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:56.528951 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:56.528964 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:56.530600 (+ 1636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260809 07:53:56.530853 10119 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:56.525915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:43345 (local address 127.28.112.62:38715)
0809 07:53:56.526038 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:56.526041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:56.526054 (+ 13us) server_negotiation.cc:407] Connection header received
0809 07:53:56.526122 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:56.526124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:56.526161 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:56.526223 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:56.526615 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.527098 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.528164 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.528366 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.528458 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:56.530746 (+ 2288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.1:43345: BlockingRecv error: recv got EOF from 127.28.112.1:43345 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 09 07:53:56 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:53:56 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:53:56.535699 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:56.531220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.1:57779)
0809 07:53:56.531365 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:56.531377 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:56.531445 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:56.531640 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:56.531642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:56.531754 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:56.531818 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:56.531822 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.532751 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.532753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:56.533236 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:56.533242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.533733 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.533735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:56.533791 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:56.534196 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:56.534206 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:56.535619 (+ 1413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87}
W20260809 07:53:56.535856 10119 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:56.531313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:57779 (local address 127.28.112.62:38715)
0809 07:53:56.531429 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:56.531433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:56.531448 (+ 15us) server_negotiation.cc:407] Connection header received
0809 07:53:56.531512 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:56.531515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:56.531556 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:56.531632 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:56.531897 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.532669 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.533319 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.533645 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.533741 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:56.535756 (+ 2015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.1:57779: BlockingRecv error: recv got EOF from 127.28.112.1:57779 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260809 07:53:56.536038 10115 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 07:53:56.626377 10122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:56.626617 10122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:56.626660 10122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:56.630585 10122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:56.630638 10122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:56.630698 10122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:53:56.635742 10122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.2.keytab
--principal=oryx/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:38243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=36339
--tserver_master_addrs=127.28.112.62:38715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:56.636795 10122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:56.637753 10122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:56.646802 10129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:56.646795 10128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:56.647022 10122 server_base.cc:1034] running on GCE node
W20260809 07:53:56.647195 10132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:56.647598 10122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:56.648263 10122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:56.651546 10122 hybrid_clock.cc:648] HybridClock initialized: now 1786262036651509 us; error 56 us; skew 500 ppm
Aug 09 07:53:56 dist-test-slave-vpvm krb5kdc[8999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262036, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:56.654165 10122 init.cc:377] Logged in from keytab as oryx/127.28.112.2@KRBTEST.COM (short username oryx)
I20260809 07:53:56.654971 10122 webserver.cc:460] Webserver started at http://127.28.112.2:36339/ using document root <none> and password file <none>
I20260809 07:53:56.655488 10122 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:56.655562 10122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:56.659194 10122 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:56.661113 10144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:56.662354 10122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:53:56.662451 10122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "a36d5f1d7f3b4236993d68b6c06d05d4"
format_stamp: "Formatted at 2026-08-09 07:53:55 on dist-test-slave-vpvm"
server_key: "e47b140e13f00cd9c271715a51b93f2f"
server_key_iv: "1abc3c9e8ee72080bd9c6e47d4c3d4c4"
server_key_version: "encryptionkey@0"
I20260809 07:53:56.662933 10122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:56.672974 10122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:56.675823 10122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:56.675990 10122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:56.676515 10122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:56.677289 10122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:56.677335 10122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:56.677377 10122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:56.677407 10122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:56.685689 10122 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:38243
I20260809 07:53:56.685715 10314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:38243 every 8 connection(s)
I20260809 07:53:56.686647 10122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:53:56.690099 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 10122
I20260809 07:53:56.690595 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:38743
--local_ip_for_outbound_sockets=127.28.112.3
--tserver_master_addrs=127.28.112.62:38715
--webserver_port=37317
--webserver_interface=127.28.112.3
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.112.3
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 07:53:56 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:53:56 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 07:53:56.697751 10119 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:56.688320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:41581 (local address 127.28.112.62:38715)
0809 07:53:56.688434 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:56.688437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:56.689396 (+ 959us) server_negotiation.cc:407] Connection header received
0809 07:53:56.690302 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:56.690306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:56.690371 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:56.690456 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:56.691622 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.692310 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.693850 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.694094 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.694169 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:56.697601 (+ 3432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.2:41581: BlockingRecv error: recv got EOF from 127.28.112.2:41581 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 07:53:56.698159 10317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:56.688634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.2:41581)
0809 07:53:56.689252 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:56.689292 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:56.690091 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:56.690607 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:56.690613 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:56.691009 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:56.691474 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:56.691488 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.692443 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.692447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:56.693746 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:56.693757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.694237 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.694241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:56.694427 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:56.694978 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:56.694999 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:56.697335 (+ 2336us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":111,"threads_started":1}
W20260809 07:53:56.698725 10315 heartbeater.cc:644] Failed to heartbeat to 127.28.112.62:38715 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.112.62:38715: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Aug 09 07:53:56 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:53:56 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:53:56.704071 10317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:56.699147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.2:40299)
0809 07:53:56.699385 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:56.699397 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:56.699457 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:56.699700 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:56.699702 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:56.699822 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:56.699908 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:56.699912 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.700861 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.700863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:56.701381 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:56.701387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.701917 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.701919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:56.702029 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:56.702422 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:56.702434 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:56.703971 (+ 1537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180}
W20260809 07:53:56.704234 10119 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:56.699276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:40299 (local address 127.28.112.62:38715)
0809 07:53:56.699404 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:56.699408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:56.699426 (+ 18us) server_negotiation.cc:407] Connection header received
0809 07:53:56.699548 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:56.699552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:56.699602 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:56.699695 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:56.700007 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.700766 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.701483 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.701819 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.701931 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:56.704120 (+ 2189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.2:40299: BlockingRecv error: recv got EOF from 127.28.112.2:40299 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 09 07:53:56 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:53:56 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:53:56.709800 10317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:56.704863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.2:50221)
0809 07:53:56.705055 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:56.705070 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:56.705159 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:56.705397 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:56.705401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:56.705567 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:56.705655 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:56.705661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.706707 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.706710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:56.707216 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:56.707225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.707756 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.707759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:56.707818 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:56.708231 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:56.708245 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:56.709717 (+ 1472us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130}
W20260809 07:53:56.710054 10315 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 07:53:56.710255 10119 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:56.704985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:50221 (local address 127.28.112.62:38715)
0809 07:53:56.705149 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:56.705153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:56.705167 (+ 14us) server_negotiation.cc:407] Connection header received
0809 07:53:56.705228 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:56.705231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:56.705270 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:56.705348 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:56.705751 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.706567 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.707311 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:56.707644 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:56.707739 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:56.709855 (+ 2116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.2:50221: BlockingRecv error: recv got EOF from 127.28.112.2:50221 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260809 07:53:56.960687 10318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:53:56.961359 10318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:53:56.961416 10318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:53:56.968631 10318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:53:56.968693 10318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:53:56.968755 10318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:53:56.974301 10318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/krb5kdc/oryx_127.28.112.3.keytab
--principal=oryx/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:38743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=37317
--tserver_master_addrs=127.28.112.62:38715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:53:56.975315 10318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:53:56.976284 10318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:53:56.986478 10325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:56.986511 10327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:53:56.986511 10329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:53:56.986687 10318 server_base.cc:1034] running on GCE node
I20260809 07:53:56.987308 10318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:53:56.987907 10318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:53:56.989061 10318 hybrid_clock.cc:648] HybridClock initialized: now 1786262036989042 us; error 31 us; skew 500 ppm
Aug 09 07:53:56 dist-test-slave-vpvm krb5kdc[8999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262036, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:53:56.991695 10318 init.cc:377] Logged in from keytab as oryx/127.28.112.3@KRBTEST.COM (short username oryx)
I20260809 07:53:56.992727 10318 webserver.cc:460] Webserver started at http://127.28.112.3:37317/ using document root <none> and password file <none>
I20260809 07:53:56.993286 10318 fs_manager.cc:359] Metadata directory not provided
I20260809 07:53:56.993366 10318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:53:56.997515 10318 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 07:53:56.999711 10343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:57.000866 10318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:53:57.001003 10318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "d85aca7b375f4863b80e2d7b7889c062"
format_stamp: "Formatted at 2026-08-09 07:53:55 on dist-test-slave-vpvm"
server_key: "e796760b4d115116530090912f91d288"
server_key_iv: "8abc2f6170f2292315bc3ec1b7e7f9e1"
server_key_version: "encryptionkey@0"
I20260809 07:53:57.001519 10318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:53:57.028236 10318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:53:57.032872 10318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:53:57.033068 10318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:53:57.033686 10318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:53:57.034564 10318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:53:57.034614 10318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:57.034672 10318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:53:57.034696 10318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:53:57.043756 10318 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:38743
I20260809 07:53:57.043785 10508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:38743 every 8 connection(s)
I20260809 07:53:57.044788 10318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
Aug 09 07:53:57 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:53:57 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:53:57.054257 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 10318
W20260809 07:53:57.055137 10119 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:57.046522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:34825 (local address 127.28.112.62:38715)
0809 07:53:57.046662 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:57.046666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:57.047684 (+ 1018us) server_negotiation.cc:407] Connection header received
0809 07:53:57.048524 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:57.048527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:57.048575 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:57.048653 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:57.049746 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.050322 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.051388 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.051611 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.051691 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:57.055008 (+ 3317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.3:34825: BlockingRecv error: recv got EOF from 127.28.112.3:34825 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 07:53:57.055713 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:57.046892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.3:34825)
0809 07:53:57.047532 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:57.047571 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:57.048315 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:57.048803 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:57.048809 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:57.049144 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:57.049615 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:57.049628 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.050414 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.050416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:57.051281 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:57.051290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.051725 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.051728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:57.051866 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:57.052354 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:57.052369 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:57.054829 (+ 2460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":117,"threads_started":1}
W20260809 07:53:57.056157 10509 heartbeater.cc:644] Failed to heartbeat to 127.28.112.62:38715 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.112.62:38715: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Aug 09 07:53:57 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:53:57 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:53:57.061797 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:57.056582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.3:51865)
0809 07:53:57.056736 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:57.056749 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:57.056815 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:57.057052 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:57.057055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:57.057212 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:57.057311 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:57.057317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.058296 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.058299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:57.059062 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:57.059071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.059536 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.059539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:57.059612 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:57.060007 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:57.060018 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:57.061712 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260809 07:53:57.062229 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:57.056677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:51865 (local address 127.28.112.62:38715)
0809 07:53:57.056870 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:57.056873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:57.056883 (+ 10us) server_negotiation.cc:407] Connection header received
0809 07:53:57.056911 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:57.056913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:57.056946 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:57.057013 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:57.057406 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.058202 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.059170 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.059432 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.059531 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:57.061854 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.3:51865: BlockingRecv error: recv got EOF from 127.28.112.3:51865 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":59,"threads_started":1}
I20260809 07:53:57.064342 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:57.055343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34930 (local address 127.28.112.62:38715)
0809 07:53:57.055441 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:57.055445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:57.055589 (+ 144us) server_negotiation.cc:407] Connection header received
0809 07:53:57.055713 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:57.055716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:57.055755 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:57.055830 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:57.056666 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.057347 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.058513 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.058841 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.058950 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:57.060215 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:53:57.060222 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:53:57.060242 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:53:57.060272 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:53:57.062152 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:57.062821 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:57.062827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:57.062834 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:57.062944 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:53:57.063233 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:53:57.063235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:53:57.063236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:53:57.063563 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:53:57.063739 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 07:53:57.064024 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 07:53:57.064242 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":39}
Aug 09 07:53:57 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:53:57 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:53:57.067728 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:57.062329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.3:33351)
0809 07:53:57.062668 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:57.062681 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:57.062744 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:57.063128 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:57.063131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:57.063292 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:57.063362 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:57.063367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.064044 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.064048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:57.064775 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:57.064785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.065182 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.065186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:57.065259 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:57.065613 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:57.065627 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:57.067580 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229}
W20260809 07:53:57.067925 10509 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 07:53:57.067960 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:57.062737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:33351 (local address 127.28.112.62:38715)
0809 07:53:57.062884 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:57.062887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:57.062897 (+ 10us) server_negotiation.cc:407] Connection header received
0809 07:53:57.062928 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:57.062930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:57.062961 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:57.063025 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:57.063468 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.063949 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.064882 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.065091 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.065172 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:57.067729 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.3:33351: BlockingRecv error: recv got EOF from 127.28.112.3:33351 (error 108)
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":82}
Aug 09 07:53:57 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:53:57 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:53:57.541944 10520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:57.536588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.1:42819)
0809 07:53:57.536883 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:57.536895 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:57.537020 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:57.537272 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:57.537274 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:57.537486 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:57.537605 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:57.537612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.538436 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.538440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:57.539032 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:57.539041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.539476 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.539480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:57.539580 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:57.540050 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:57.540063 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:57.541829 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":90,"threads_started":1}
W20260809 07:53:57.542124 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:57.536652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:42819 (local address 127.28.112.62:38715)
0809 07:53:57.536838 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:57.536842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:57.536976 (+ 134us) server_negotiation.cc:407] Connection header received
0809 07:53:57.537113 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:57.537116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:57.537169 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:57.537236 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:57.537726 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.538321 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.539148 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.539363 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.539460 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:57.542008 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.1:42819: BlockingRecv error: recv got EOF from 127.28.112.1:42819 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 07:53:57 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:53:57 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:53:57.716101 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:57.710583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.2:57139)
0809 07:53:57.710879 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:57.710904 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:57.711035 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:57.711296 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:57.711299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:57.711510 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:57.711616 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:57.711624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.712403 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.712408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:57.712994 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:57.713004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.713519 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.713523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:57.713609 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:57.714149 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:57.714164 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:57.715959 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":83,"threads_started":1}
W20260809 07:53:57.716228 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:57.710703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:57139 (local address 127.28.112.62:38715)
0809 07:53:57.710870 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:57.710875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:57.710988 (+ 113us) server_negotiation.cc:407] Connection header received
0809 07:53:57.711127 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:57.711131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:57.711185 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:57.711259 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:57.711739 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.712283 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.713149 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:57.713377 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:57.713505 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:57.716113 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.2:57139: BlockingRecv error: recv got EOF from 127.28.112.2:57139 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 07:53:58 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:53:58 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:53:58.073624 10523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:58.068523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.3:43087)
0809 07:53:58.068802 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:58.068817 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:58.068919 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:58.069220 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:58.069222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:58.069388 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:58.069482 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:58.069486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:58.070244 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:58.070246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:58.070800 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:58.070806 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:58.071244 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:58.071247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:58.071303 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:58.071770 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:58.071780 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:58.073505 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
W20260809 07:53:58.073773 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:58.068546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:43087 (local address 127.28.112.62:38715)
0809 07:53:58.068687 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:58.068691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:58.068887 (+ 196us) server_negotiation.cc:407] Connection header received
0809 07:53:58.069072 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:58.069075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:58.069126 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:58.069206 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:58.069583 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:58.070163 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:58.070904 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:58.071153 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:58.071243 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:58.073651 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.3:43087: BlockingRecv error: recv got EOF from 127.28.112.3:43087 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 09 07:53:58 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:53:58 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:53:58.547614 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:58.542673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.1:55063)
0809 07:53:58.542933 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:58.542946 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:58.543049 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:58.543276 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:58.543278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:58.543462 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:58.543561 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:58.543567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:58.544243 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:58.544246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:58.544813 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:58.544822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:58.545172 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:58.545174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:58.545235 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:58.545717 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:58.545729 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:58.547483 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":70,"threads_started":1}
W20260809 07:53:58.547710 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:58.542732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:55063 (local address 127.28.112.62:38715)
0809 07:53:58.542849 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:58.542853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:58.543006 (+ 153us) server_negotiation.cc:407] Connection header received
0809 07:53:58.543128 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:58.543130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:58.543190 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:58.543248 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:58.543646 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:58.544151 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:58.544897 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:58.545095 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:58.545162 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:58.547613 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.1:55063: BlockingRecv error: recv got EOF from 127.28.112.1:55063 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 09 07:53:58 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:53:58 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:53:58.722052 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:58.716935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.2:42309)
0809 07:53:58.717208 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:58.717222 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:58.717332 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:58.717550 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:58.717552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:58.717744 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:58.717856 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:58.717863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:58.718588 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:58.718596 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:58.719166 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:58.719175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:58.719548 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:58.719551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:58.719619 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:58.720141 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:58.720155 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:58.721870 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
W20260809 07:53:58.722147 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:58.716993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:42309 (local address 127.28.112.62:38715)
0809 07:53:58.717114 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:58.717118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:58.717287 (+ 169us) server_negotiation.cc:407] Connection header received
0809 07:53:58.717401 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:58.717403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:58.717444 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:58.717508 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:58.717978 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:58.718489 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:58.719263 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:58.719461 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:58.719524 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:58.722049 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.2:42309: BlockingRecv error: recv got EOF from 127.28.112.2:42309 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 09 07:53:59 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:53:59 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:53:59.082432 10528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:59.074349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.3:46971)
0809 07:53:59.074636 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:59.074650 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:59.074756 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:59.074970 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:59.074973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:59.075144 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:59.075241 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:59.075246 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:59.076097 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:59.076101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:59.076867 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:59.076885 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:59.077313 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:59.077316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:59.077384 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:59.080429 (+ 3045us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:59.080444 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:59.082285 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
W20260809 07:53:59.082559 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:59.074512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:46971 (local address 127.28.112.62:38715)
0809 07:53:59.074658 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:59.074661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:59.074703 (+ 42us) server_negotiation.cc:407] Connection header received
0809 07:53:59.074830 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:59.074832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:59.074872 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:59.074936 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:59.075337 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:59.075961 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:59.076990 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:59.077201 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:59.077282 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:59.082449 (+ 5167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.3:46971: BlockingRecv error: recv got EOF from 127.28.112.3:46971 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 07:53:59 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:53:59 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:53:59.553619 10530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:59.548445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.1:54893)
0809 07:53:59.548699 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:59.548713 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:59.548820 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:59.549039 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:59.549042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:59.549219 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:59.549318 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:59.549323 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:59.550019 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:59.550022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:59.550600 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:59.550617 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:59.550975 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:59.550978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:59.551041 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:59.551570 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:59.551582 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:59.553448 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
W20260809 07:53:59.553714 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:59.548527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:54893 (local address 127.28.112.62:38715)
0809 07:53:59.548705 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:59.548710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:59.548779 (+ 69us) server_negotiation.cc:407] Connection header received
0809 07:53:59.548893 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:59.548896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:59.548947 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:59.549012 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:59.549401 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:59.549903 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:59.550694 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:59.550890 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:59.550957 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:59.553608 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.1:54893: BlockingRecv error: recv got EOF from 127.28.112.1:54893 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 09 07:53:59 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:53:59 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:53:59.727914 10533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:53:59.722805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.2:40583)
0809 07:53:59.723084 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0809 07:53:59.723096 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:53:59.723227 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:53:59.723447 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:53:59.723450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:53:59.723627 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:53:59.723727 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:59.723733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:59.724377 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:59.724379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:59.724920 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:53:59.724927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:53:59.725262 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:53:59.725264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:53:59.725322 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:59.725864 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:53:59.725876 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:53:59.727764 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":93,"threads_started":1}
W20260809 07:53:59.727977 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:53:59.722925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:40583 (local address 127.28.112.62:38715)
0809 07:53:59.723068 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 07:53:59.723071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:53:59.723210 (+ 139us) server_negotiation.cc:407] Connection header received
0809 07:53:59.723311 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:53:59.723313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:53:59.723359 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:53:59.723417 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:53:59.723810 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:59.724304 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:59.724995 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:53:59.725190 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:53:59.725251 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:53:59.727879 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.2:40583: BlockingRecv error: recv got EOF from 127.28.112.2:40583 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 09 07:54:00 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:00 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:00.088184 10535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:00.083266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.3:49159)
0809 07:54:00.083512 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:00.083525 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:00.083624 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:00.083841 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:00.083843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:00.084018 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:00.084116 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:00.084121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:00.084793 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:00.084795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:00.085336 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:00.085343 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:00.085707 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:00.085709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:00.085765 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:00.086233 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:00.086244 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:00.088022 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
W20260809 07:54:00.088253 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:00.083310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:49159 (local address 127.28.112.62:38715)
0809 07:54:00.083439 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:00.083443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:00.083584 (+ 141us) server_negotiation.cc:407] Connection header received
0809 07:54:00.083704 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:00.083706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:00.083749 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:00.083815 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:00.084203 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:00.084708 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:00.085417 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:00.085627 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:00.085699 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:00.088154 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.3:49159: BlockingRecv error: recv got EOF from 127.28.112.3:49159 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 09 07:54:00 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:00 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:00.559468 10536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:00.554400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.1:60633)
0809 07:54:00.554664 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:00.554678 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:00.554785 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:00.555000 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:00.555002 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:00.555179 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:00.555278 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:00.555283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:00.555965 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:00.555968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:00.556527 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:00.556534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:00.556894 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:00.556896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:00.556964 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:00.557443 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:00.557455 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:00.559314 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":72,"threads_started":1}
W20260809 07:54:00.559556 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:00.554532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:60633 (local address 127.28.112.62:38715)
0809 07:54:00.554692 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:00.554695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:00.554731 (+ 36us) server_negotiation.cc:407] Connection header received
0809 07:54:00.554860 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:00.554862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:00.554905 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:00.554970 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:00.555362 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:00.555862 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:00.556613 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:00.556821 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:00.556886 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:00.559457 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.1:60633: BlockingRecv error: recv got EOF from 127.28.112.1:60633 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 09 07:54:00 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:00 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:00.734139 10537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:00.728864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.2:51369)
0809 07:54:00.729157 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:00.729171 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:00.729301 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:00.729574 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:00.729578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:00.729773 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:00.729877 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:00.729882 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:00.730695 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:00.730698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:00.731257 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:00.731265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:00.731655 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:00.731657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:00.731715 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:00.732190 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:00.732202 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:00.734022 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":103,"threads_started":1}
W20260809 07:54:00.734274 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:00.728944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:51369 (local address 127.28.112.62:38715)
0809 07:54:00.729100 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:00.729105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:00.729260 (+ 155us) server_negotiation.cc:407] Connection header received
0809 07:54:00.729385 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:00.729388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:00.729448 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:00.729550 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:00.729996 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:00.730569 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:00.731350 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:00.731570 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:00.731652 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:00.734145 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.2:51369: BlockingRecv error: recv got EOF from 127.28.112.2:51369 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 09 07:54:01 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:01 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:01.094198 10538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:01.089015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.3:52387)
0809 07:54:01.089279 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:01.089293 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:01.089395 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:01.089605 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:01.089607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:01.089779 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:01.089897 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:01.089902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:01.090779 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:01.090783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:01.091520 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:01.091531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:01.091920 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:01.091922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:01.091978 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:01.092411 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:01.092422 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:01.094070 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
W20260809 07:54:01.094280 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:01.089048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:52387 (local address 127.28.112.62:38715)
0809 07:54:01.089194 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:01.089198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:01.089357 (+ 159us) server_negotiation.cc:407] Connection header received
0809 07:54:01.089467 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:01.089470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:01.089513 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:01.089578 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:01.090018 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:01.090665 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:01.091625 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:01.091833 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:01.091913 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:01.094180 (+ 2267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.3:52387: BlockingRecv error: recv got EOF from 127.28.112.3:52387 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 09 07:54:01 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:01 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:01.565362 10539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:01.560252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.1:55179)
0809 07:54:01.560530 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:01.560544 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:01.560647 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:01.560907 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:01.560909 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:01.561106 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:01.561206 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:01.561212 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:01.561902 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:01.561905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:01.562547 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:01.562558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:01.562930 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:01.562934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:01.563001 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:01.563523 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:01.563535 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:01.565247 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":75,"threads_started":1}
W20260809 07:54:01.565491 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:01.560347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:55179 (local address 127.28.112.62:38715)
0809 07:54:01.560502 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:01.560505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:01.560611 (+ 106us) server_negotiation.cc:407] Connection header received
0809 07:54:01.560719 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:01.560722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:01.560764 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:01.560827 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:01.561301 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:01.561800 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:01.562630 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:01.562831 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:01.562899 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:01.565374 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.1:55179: BlockingRecv error: recv got EOF from 127.28.112.1:55179 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 07:54:01 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:01 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:01.740255 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:01.734862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.2:40803)
0809 07:54:01.735144 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:01.735159 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:01.735268 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:01.735550 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:01.735554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:01.735754 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:01.735845 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:01.735852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:01.736591 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:01.736595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:01.737184 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:01.737194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:01.737630 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:01.737634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:01.737718 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:01.738266 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:01.738278 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:01.740089 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":108,"threads_started":1}
W20260809 07:54:01.740298 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:01.734997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:40803 (local address 127.28.112.62:38715)
0809 07:54:01.735128 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:01.735132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:01.735223 (+ 91us) server_negotiation.cc:407] Connection header received
0809 07:54:01.735364 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:01.735367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:01.735415 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:01.735505 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:01.735957 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:01.736490 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:01.737292 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:01.737530 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:01.737615 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:01.740201 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.2:40803: BlockingRecv error: recv got EOF from 127.28.112.2:40803 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 09 07:54:02 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:02 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:02.099952 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:02.094972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.3:35579)
0809 07:54:02.095236 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:02.095249 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:02.095347 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:02.095575 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:02.095577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:02.095760 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:02.095853 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:02.095858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:02.096533 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:02.096535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:02.097079 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:02.097085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:02.097431 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:02.097433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:02.097489 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:02.097997 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:02.098009 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:02.099835 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":86,"threads_started":1}
W20260809 07:54:02.100142 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:02.095033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:35579 (local address 127.28.112.62:38715)
0809 07:54:02.095185 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:02.095191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:02.095315 (+ 124us) server_negotiation.cc:407] Connection header received
0809 07:54:02.095428 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:02.095430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:02.095477 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:02.095553 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:02.095938 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:02.096455 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:02.097156 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:02.097355 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:02.097431 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:02.099998 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.3:35579: BlockingRecv error: recv got EOF from 127.28.112.3:35579 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 09 07:54:02 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:02 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:02.571534 10545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:02.566179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.1:46699)
0809 07:54:02.566491 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:02.566506 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:02.566611 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:02.566853 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:02.566870 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:02.567053 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:02.567152 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:02.567158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:02.567907 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:02.567910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:02.568551 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:02.568561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:02.568962 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:02.568965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:02.569034 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:02.569558 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:02.569569 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:02.571378 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":92,"threads_started":1}
W20260809 07:54:02.571653 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:02.566228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:46699 (local address 127.28.112.62:38715)
0809 07:54:02.566443 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:02.566449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:02.566575 (+ 126us) server_negotiation.cc:407] Connection header received
0809 07:54:02.566691 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:02.566693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:02.566739 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:02.566845 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:02.567262 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:02.567808 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:02.568656 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:02.568873 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:02.568957 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:02.571552 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.1:46699: BlockingRecv error: recv got EOF from 127.28.112.1:46699 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 09 07:54:02 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:02 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:02.746697 10546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:02.741074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.2:51797)
0809 07:54:02.741360 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:02.741378 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:02.741514 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:02.741780 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:02.741785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:02.742045 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:02.742150 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:02.742157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:02.742942 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:02.742945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:02.743511 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:02.743520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:02.743973 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:02.743978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:02.744065 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:02.744570 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:02.744583 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:02.746548 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":85,"threads_started":1}
W20260809 07:54:02.746775 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:02.741179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:51797 (local address 127.28.112.62:38715)
0809 07:54:02.741343 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:02.741348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:02.741460 (+ 112us) server_negotiation.cc:407] Connection header received
0809 07:54:02.741610 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:02.741614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:02.741665 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:02.741744 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:02.742270 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:02.742810 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:02.743630 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:02.743862 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:02.743969 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:02.746675 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.2:51797: BlockingRecv error: recv got EOF from 127.28.112.2:51797 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 09 07:54:03 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:03 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:03.106076 10547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:03.100756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.3:54175)
0809 07:54:03.101022 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:03.101034 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:03.101144 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:03.101375 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:03.101378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:03.101573 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:03.101679 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:03.101685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:03.102484 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:03.102488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:03.103036 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:03.103045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:03.103483 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:03.103487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:03.103570 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:03.104026 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:03.104037 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:03.105936 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":86,"threads_started":1}
W20260809 07:54:03.106220 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:03.100818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:54175 (local address 127.28.112.62:38715)
0809 07:54:03.100960 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:03.100965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:03.101095 (+ 130us) server_negotiation.cc:407] Connection header received
0809 07:54:03.101228 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:03.101231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:03.101276 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:03.101341 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:03.101788 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:03.102377 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:03.103166 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:03.103380 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:03.103471 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:03.106088 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.3:54175: BlockingRecv error: recv got EOF from 127.28.112.3:54175 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 09 07:54:03 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:03 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:03.577930 10548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:03.572419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.1:49467)
0809 07:54:03.572752 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:03.572765 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:03.572879 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:03.573186 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:03.573189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:03.573431 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:03.573561 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:03.573571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:03.574478 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:03.574481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:03.575046 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:03.575055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:03.575412 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:03.575416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:03.575476 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:03.575941 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:03.575952 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:03.577809 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":79,"threads_started":1}
W20260809 07:54:03.578073 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:03.572517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:49467 (local address 127.28.112.62:38715)
0809 07:54:03.572675 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:03.572680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:03.572833 (+ 153us) server_negotiation.cc:407] Connection header received
0809 07:54:03.572990 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:03.572993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:03.573036 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:03.573126 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:03.573689 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:03.574371 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:03.575136 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:03.575338 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:03.575404 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:03.577936 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.1:49467: BlockingRecv error: recv got EOF from 127.28.112.1:49467 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 09 07:54:03 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:03 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:03.752825 10549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:03.747543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.2:51325)
0809 07:54:03.747900 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:03.747913 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:03.748025 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:03.748265 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:03.748268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:03.748508 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:03.748604 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:03.748610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:03.749351 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:03.749355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:03.749901 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:03.749909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:03.750364 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:03.750367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:03.750467 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:03.750928 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:03.750942 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:03.752709 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":87,"threads_started":1}
W20260809 07:54:03.752971 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:03.747661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:51325 (local address 127.28.112.62:38715)
0809 07:54:03.747814 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:03.747819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:03.747984 (+ 165us) server_negotiation.cc:407] Connection header received
0809 07:54:03.748111 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:03.748115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:03.748161 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:03.748226 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:03.748718 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:03.749241 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:03.750051 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:03.750261 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:03.750349 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:03.752844 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.2:51325: BlockingRecv error: recv got EOF from 127.28.112.2:51325 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 07:54:04 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:04 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:04.115475 10550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:04.106747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.3:59237)
0809 07:54:04.107046 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:04.107059 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:04.107158 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:04.107339 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:04.107342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:04.107530 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:04.107624 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:04.107630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:04.108331 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:04.108334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:04.108911 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:04.108921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:04.109375 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:04.109379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:04.109496 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:04.113555 (+ 4059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:04.113568 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:04.115352 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":71,"threads_started":1}
W20260809 07:54:04.115597 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:04.106797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:59237 (local address 127.28.112.62:38715)
0809 07:54:04.106923 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:04.106926 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:04.107122 (+ 196us) server_negotiation.cc:407] Connection header received
0809 07:54:04.107221 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:04.107223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:04.107264 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:04.107326 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:04.107705 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:04.108211 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:04.109015 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:04.109247 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:04.109321 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:04.115479 (+ 6158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.3:59237: BlockingRecv error: recv got EOF from 127.28.112.3:59237 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 09 07:54:04 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:04 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:04.584467 10551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:04.578762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.1:41035)
0809 07:54:04.579075 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:04.579093 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:04.579235 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:04.579509 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:04.579513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:04.579727 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:04.579826 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:04.579833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:04.580604 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:04.580616 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:04.581197 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:04.581206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:04.581664 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:04.581668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:04.581760 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:04.582256 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:04.582269 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:04.584351 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":84,"threads_started":1}
W20260809 07:54:04.584631 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:04.578930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:41035 (local address 127.28.112.62:38715)
0809 07:54:04.579084 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:04.579090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:04.579171 (+ 81us) server_negotiation.cc:407] Connection header received
0809 07:54:04.579344 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:04.579346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:04.579392 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:04.579470 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:04.579956 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:04.580479 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:04.581318 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:04.581543 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:04.581637 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:04.584499 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.1:41035: BlockingRecv error: recv got EOF from 127.28.112.1:41035 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 09 07:54:04 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:04 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:04.758579 10552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:04.753571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.2:43977)
0809 07:54:04.753814 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:04.753828 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:04.753926 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:04.754141 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:04.754143 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:04.754322 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:04.754418 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:04.754423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:04.755230 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:04.755232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:04.755802 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:04.755811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:04.756237 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:04.756239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:04.756293 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:04.756729 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:04.756740 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:04.758457 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":77,"threads_started":1}
W20260809 07:54:04.758675 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:04.753629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:43977 (local address 127.28.112.62:38715)
0809 07:54:04.753750 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:04.753754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:04.753889 (+ 135us) server_negotiation.cc:407] Connection header received
0809 07:54:04.754013 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:04.754015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:04.754058 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:04.754128 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:04.754499 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:04.755147 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:04.755922 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:04.756147 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:04.756225 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:04.758580 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.2:43977: BlockingRecv error: recv got EOF from 127.28.112.2:43977 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 09 07:54:05 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:05 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:05.121479 10554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:05.116257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.3:41737)
0809 07:54:05.116557 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:05.116572 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:05.116693 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:05.116949 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:05.116951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:05.117146 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:05.117244 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:05.117250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:05.117996 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:05.118000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:05.118610 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:05.118621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:05.119150 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:05.119154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:05.119241 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:05.119733 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:05.119746 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:05.121365 (+ 1619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":92,"threads_started":1}
W20260809 07:54:05.121604 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:05.116372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:41737 (local address 127.28.112.62:38715)
0809 07:54:05.116546 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:05.116551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:05.116636 (+ 85us) server_negotiation.cc:407] Connection header received
0809 07:54:05.116796 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:05.116799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:05.116845 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:05.116908 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:05.117356 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:05.117868 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:05.118731 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:05.119026 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:05.119136 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:05.121497 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.3:41737: BlockingRecv error: recv got EOF from 127.28.112.3:41737 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 09 07:54:05 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:05 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:05.590082 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:05.585124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.1:37233)
0809 07:54:05.585392 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:05.585405 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:05.585503 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:05.585703 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:05.585705 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:05.585878 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:05.586012 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:05.586020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:05.586700 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:05.586702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:05.587237 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:05.587243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:05.587661 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:05.587663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:05.587720 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:05.588151 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:05.588162 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:05.589934 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
W20260809 07:54:05.590270 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:05.585207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:37233 (local address 127.28.112.62:38715)
0809 07:54:05.585336 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:05.585340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:05.585484 (+ 144us) server_negotiation.cc:407] Connection header received
0809 07:54:05.585582 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:05.585585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:05.585631 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:05.585689 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:05.586100 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:05.586621 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:05.587337 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:05.587570 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:05.587648 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:05.590155 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.1:37233: BlockingRecv error: recv got EOF from 127.28.112.1:37233 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 09 07:54:05 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:05 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:05.764611 10556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:05.759271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.2:53219)
0809 07:54:05.759551 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:05.759565 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:05.759689 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:05.759927 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:05.759930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:05.760126 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:05.760221 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:05.760227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:05.760948 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:05.760953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:05.761493 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:05.761500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:05.761923 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:05.761927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:05.762044 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:05.762506 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:05.762516 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:05.764421 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":90,"threads_started":1}
W20260809 07:54:05.764631 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:05.759358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:53219 (local address 127.28.112.62:38715)
0809 07:54:05.759517 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:05.759522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:05.759630 (+ 108us) server_negotiation.cc:407] Connection header received
0809 07:54:05.759777 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:05.759780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:05.759824 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:05.759890 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:05.760337 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:05.760844 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:05.761606 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:05.761821 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:05.761908 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:05.764540 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.2:53219: BlockingRecv error: recv got EOF from 127.28.112.2:53219 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 09 07:54:06 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:06 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:06.127723 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:06.122228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.3:37447)
0809 07:54:06.122522 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:06.122536 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:06.122641 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:06.122891 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:06.122894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:06.123082 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:06.123198 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:06.123204 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:06.123894 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:06.123897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:06.124497 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:06.124506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:06.124888 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:06.124891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:06.124966 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:06.125493 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:06.125507 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:06.127546 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":118,"threads_started":1}
W20260809 07:54:06.127796 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:06.122290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:37447 (local address 127.28.112.62:38715)
0809 07:54:06.122420 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:06.122423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:06.122599 (+ 176us) server_negotiation.cc:407] Connection header received
0809 07:54:06.122721 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:06.122723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:06.122766 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:06.122824 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:06.123292 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:06.123790 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:06.124587 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:06.124792 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:06.124860 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:06.127697 (+ 2837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.3:37447: BlockingRecv error: recv got EOF from 127.28.112.3:37447 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 09 07:54:06 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:06 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:06.595883 10558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:06.590820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.1:55639)
0809 07:54:06.591086 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:06.591099 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:06.591195 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:06.591394 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:06.591396 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:06.591579 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:06.591677 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:06.591682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:06.592349 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:06.592353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:06.592910 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:06.592918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:06.593355 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:06.593359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:06.593437 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:06.593991 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:06.594002 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:06.595719 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":73,"threads_started":1}
W20260809 07:54:06.595945 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:06.590895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:55639 (local address 127.28.112.62:38715)
0809 07:54:06.591018 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:06.591021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:06.591157 (+ 136us) server_negotiation.cc:407] Connection header received
0809 07:54:06.591260 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:06.591263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:06.591301 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:06.591368 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:06.591761 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:06.592262 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:06.593042 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:06.593253 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:06.593341 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:06.595846 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.1:55639: BlockingRecv error: recv got EOF from 127.28.112.1:55639 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 09 07:54:06 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:06 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:06.770640 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:06.765378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.2:51713)
0809 07:54:06.765674 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:06.765690 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:06.765796 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:06.766015 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:06.766017 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:06.766234 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:06.766352 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:06.766360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:06.767166 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:06.767168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:06.767777 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:06.767786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:06.768185 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:06.768187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:06.768242 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:06.768775 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:06.768786 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:06.770510 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":77,"threads_started":1}
W20260809 07:54:06.770732 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:06.765453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:51713 (local address 127.28.112.62:38715)
0809 07:54:06.765609 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:06.765614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:06.765756 (+ 142us) server_negotiation.cc:407] Connection header received
0809 07:54:06.765863 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:06.765865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:06.765904 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:06.765981 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:06.766454 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:06.767065 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:06.767876 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:06.768100 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:06.768185 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:06.770632 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.2:51713: BlockingRecv error: recv got EOF from 127.28.112.2:51713 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 09 07:54:07 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:07 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:07.133785 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:07.128585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.3:37487)
0809 07:54:07.128859 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:07.128872 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:07.128960 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:07.129205 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:07.129207 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:07.129383 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:07.129481 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:07.129485 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:07.130238 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:07.130241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:07.130789 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:07.130796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:07.131235 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:07.131237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:07.131292 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:07.131778 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:07.131790 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:07.133641 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
W20260809 07:54:07.133883 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:07.128708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:37487 (local address 127.28.112.62:38715)
0809 07:54:07.128917 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:07.128921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:07.128941 (+ 20us) server_negotiation.cc:407] Connection header received
0809 07:54:07.129056 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:07.129059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:07.129105 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:07.129188 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:07.129577 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:07.130152 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:07.130919 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:07.131137 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:07.131224 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:07.133770 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.3:37487: BlockingRecv error: recv got EOF from 127.28.112.3:37487 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 09 07:54:07 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:07 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:07.601542 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:07.596626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.1:36273)
0809 07:54:07.596892 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:07.596906 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:07.597009 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:07.597212 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:07.597214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:07.597418 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:07.597540 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:07.597549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:07.598267 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:07.598270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:07.598813 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:07.598821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:07.599207 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:07.599210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:07.599265 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:07.599742 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:07.599754 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:07.601429 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":87,"threads_started":1}
W20260809 07:54:07.601723 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:07.596721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:36273 (local address 127.28.112.62:38715)
0809 07:54:07.596872 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:07.596877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:07.596970 (+ 93us) server_negotiation.cc:407] Connection header received
0809 07:54:07.597078 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:07.597080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:07.597122 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:07.597184 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:07.597660 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:07.598189 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:07.598914 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:07.599127 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:07.599201 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:07.601609 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.1:36273: BlockingRecv error: recv got EOF from 127.28.112.1:36273 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 07:54:07 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:07 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:07.776984 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:07.771403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.2:34801)
0809 07:54:07.771691 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:07.771708 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:07.771838 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:07.772117 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:07.772121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:07.772332 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:07.772442 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:07.772450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:07.773227 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:07.773231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:07.773979 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:07.773992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:07.774542 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:07.774546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:07.774612 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:07.775074 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:07.775086 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:07.776853 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":90,"threads_started":1}
W20260809 07:54:07.777091 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:07.771530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:34801 (local address 127.28.112.62:38715)
0809 07:54:07.771675 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:07.771680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:07.771784 (+ 104us) server_negotiation.cc:407] Connection header received
0809 07:54:07.771948 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:07.771952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:07.772002 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:07.772077 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:07.772569 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:07.773106 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:07.774102 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:07.774420 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:07.774511 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:07.776977 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.2:34801: BlockingRecv error: recv got EOF from 127.28.112.2:34801 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 09 07:54:08 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:08 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:08.139822 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:08.134501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.3:56921)
0809 07:54:08.134835 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:08.134849 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:08.134950 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:08.135202 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:08.135204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:08.135430 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:08.135556 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:08.135562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:08.136256 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:08.136258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:08.136868 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:08.136878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:08.137281 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:08.137284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:08.137353 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:08.137865 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:08.137877 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:08.139670 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":82,"threads_started":1}
W20260809 07:54:08.139896 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:08.134621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:56921 (local address 127.28.112.62:38715)
0809 07:54:08.134772 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:08.134777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:08.134914 (+ 137us) server_negotiation.cc:407] Connection header received
0809 07:54:08.135030 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:08.135033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:08.135086 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:08.135167 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:08.135655 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:08.136175 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:08.136965 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:08.137186 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:08.137259 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:08.139802 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.3:56921: BlockingRecv error: recv got EOF from 127.28.112.3:56921 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 09 07:54:08 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:08 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:08.607421 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:08.602275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.1:55569)
0809 07:54:08.602528 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:08.602542 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:08.602644 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:08.602839 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:08.602841 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:08.603017 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:08.603115 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:08.603121 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:08.603808 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:08.603811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:08.604374 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:08.604381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:08.604803 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:08.604807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:08.604919 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:08.605490 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:08.605503 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:08.607301 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
W20260809 07:54:08.607534 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:08.602327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:55569 (local address 127.28.112.62:38715)
0809 07:54:08.602455 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:08.602460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:08.602599 (+ 139us) server_negotiation.cc:407] Connection header received
0809 07:54:08.602707 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:08.602709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:08.602749 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:08.602814 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:08.603216 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:08.603718 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:08.604474 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:08.604685 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:08.604775 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:08.607431 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.1:55569: BlockingRecv error: recv got EOF from 127.28.112.1:55569 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 09 07:54:08 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:08 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:08.782848 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:08.777677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.2:33305)
0809 07:54:08.777977 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:08.777991 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:08.778103 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:08.778347 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:08.778350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:08.778545 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:08.778640 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:08.778648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:08.779373 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:08.779377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:08.779921 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:08.779929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:08.780349 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:08.780352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:08.780432 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:08.780890 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:08.780901 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:08.782732 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":89,"threads_started":1}
W20260809 07:54:08.782966 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:08.777773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:33305 (local address 127.28.112.62:38715)
0809 07:54:08.777928 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:08.777932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:08.778055 (+ 123us) server_negotiation.cc:407] Connection header received
0809 07:54:08.778198 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:08.778201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:08.778248 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:08.778313 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:08.778755 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:08.779268 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:08.780035 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:08.780249 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:08.780337 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:08.782855 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.2:33305: BlockingRecv error: recv got EOF from 127.28.112.2:33305 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 07:54:09 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:09 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:09.149534 10571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:09.140615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.3:40325)
0809 07:54:09.140896 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:09.140909 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:09.141009 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:09.141220 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:09.141222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:09.141399 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:09.141492 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:09.141497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:09.142229 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:09.142231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:09.142780 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:09.142787 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:09.143214 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:09.143216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:09.143274 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:09.147624 (+ 4350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:09.147639 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:09.149415 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":73,"threads_started":1}
W20260809 07:54:09.149667 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:09.140697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:40325 (local address 127.28.112.62:38715)
0809 07:54:09.140857 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:09.140862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:09.141002 (+ 140us) server_negotiation.cc:407] Connection header received
0809 07:54:09.141076 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:09.141079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:09.141125 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:09.141199 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:09.141613 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:09.142152 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:09.142889 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:09.143111 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:09.143201 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:09.149546 (+ 6345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.3:40325: BlockingRecv error: recv got EOF from 127.28.112.3:40325 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 09 07:54:09 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:09 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:09.613834 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:09.608278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.1:49647)
0809 07:54:09.608552 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:09.608564 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:09.608687 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:09.608922 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:09.608925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:09.609131 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:09.609232 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:09.609238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:09.610011 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:09.610015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:09.610713 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:09.610725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:09.611254 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:09.611257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:09.611325 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:09.611817 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:09.611830 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:09.613714 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":87,"threads_started":1}
W20260809 07:54:09.614044 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:09.608352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:49647 (local address 127.28.112.62:38715)
0809 07:54:09.608485 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:09.608489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:09.608627 (+ 138us) server_negotiation.cc:407] Connection header received
0809 07:54:09.608769 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:09.608772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:09.608820 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:09.608885 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:09.609342 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:09.609884 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:09.610832 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:09.611138 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:09.611215 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:09.613875 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.1:49647: BlockingRecv error: recv got EOF from 127.28.112.1:49647 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 09 07:54:09 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:09 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:09.788950 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:09.783616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.2:41663)
0809 07:54:09.783878 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:09.783891 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:09.783992 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:09.784217 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:09.784219 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:09.784403 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:09.784495 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:09.784501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:09.785228 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:09.785230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:09.785784 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:09.785791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:09.786347 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:09.786351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:09.786453 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:09.786913 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:09.786925 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:09.788762 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
W20260809 07:54:09.789019 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:09.783733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:41663 (local address 127.28.112.62:38715)
0809 07:54:09.783852 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:09.783855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:09.783980 (+ 125us) server_negotiation.cc:407] Connection header received
0809 07:54:09.784068 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:09.784071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:09.784121 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:09.784200 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:09.784599 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:09.785145 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:09.785887 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:09.786215 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:09.786321 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:09.788898 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.2:41663: BlockingRecv error: recv got EOF from 127.28.112.2:41663 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 09 07:54:10 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:10 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:10.155638 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:10.150235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.3:49255)
0809 07:54:10.150551 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:10.150565 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:10.150671 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:10.150874 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:10.150876 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:10.151073 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:10.151169 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:10.151174 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:10.151850 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:10.151853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:10.152412 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:10.152421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:10.152889 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:10.152892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:10.152966 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:10.153496 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:10.153509 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:10.155524 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":74,"threads_started":1}
W20260809 07:54:10.155757 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:10.150308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:49255 (local address 127.28.112.62:38715)
0809 07:54:10.150444 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:10.150449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:10.150630 (+ 181us) server_negotiation.cc:407] Connection header received
0809 07:54:10.150742 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:10.150744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:10.150787 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:10.150852 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:10.151255 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:10.151750 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:10.152541 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:10.152759 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:10.152834 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:10.155641 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.3:49255: BlockingRecv error: recv got EOF from 127.28.112.3:49255 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 09 07:54:10 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:10 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:10.619726 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:10.614600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.1:36863)
0809 07:54:10.614859 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:10.614873 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:10.614979 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:10.615189 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:10.615191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:10.615367 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:10.615465 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:10.615469 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:10.616145 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:10.616147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:10.616689 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:10.616695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:10.617112 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:10.617115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:10.617206 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:10.617729 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:10.617741 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:10.619602 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
W20260809 07:54:10.619822 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:10.614671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:36863 (local address 127.28.112.62:38715)
0809 07:54:10.614889 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:10.614892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:10.614937 (+ 45us) server_negotiation.cc:407] Connection header received
0809 07:54:10.615061 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:10.615064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:10.615106 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:10.615161 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:10.615552 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:10.616062 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:10.616777 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:10.616995 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:10.617093 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:10.619726 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.1:36863: BlockingRecv error: recv got EOF from 127.28.112.1:36863 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
Aug 09 07:54:10 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:10 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:10.796276 10581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:10.789889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.2:43373)
0809 07:54:10.790436 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:10.790453 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:10.790616 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:10.791017 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:10.791020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:10.791236 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:10.791342 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:10.791350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:10.792117 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:10.792122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:10.792988 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:10.793000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:10.793436 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:10.793439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:10.793524 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:10.794045 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:10.794059 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:10.796107 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":128,"threads_started":1}
W20260809 07:54:10.796334 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:10.790007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:43373 (local address 127.28.112.62:38715)
0809 07:54:10.790156 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:10.790161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:10.790525 (+ 364us) server_negotiation.cc:407] Connection header received
0809 07:54:10.790740 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:10.790744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:10.790799 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:10.790851 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:10.791480 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:10.792002 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:10.793114 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:10.793326 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:10.793407 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:10.796236 (+ 2829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.2:43373: BlockingRecv error: recv got EOF from 127.28.112.2:43373 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 07:54:11 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:11 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.3@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:11.161582 10582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:11.156482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.3:47977)
0809 07:54:11.156738 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:11.156751 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:11.156850 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:11.157070 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:11.157073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:11.157255 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:11.157373 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:11.157379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:11.158131 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:11.158134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:11.158732 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:11.158741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:11.159174 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:11.159176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:11.159242 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:11.159680 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:11.159691 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:11.161450 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":79,"threads_started":1}
W20260809 07:54:11.161667 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:11.156550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:47977 (local address 127.28.112.62:38715)
0809 07:54:11.156680 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:11.156683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:11.156809 (+ 126us) server_negotiation.cc:407] Connection header received
0809 07:54:11.156928 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:11.156930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:11.156979 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:11.157042 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:11.157483 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:11.158031 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:11.158852 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:11.159077 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:11.159156 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:11.161573 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.3:47977: BlockingRecv error: recv got EOF from 127.28.112.3:47977 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 09 07:54:11 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:11 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.1@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:11.626016 10584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:11.620577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.1:37155)
0809 07:54:11.620895 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:11.620907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:11.621020 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:11.621263 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:11.621266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:11.621466 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:11.621589 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:11.621595 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:11.622388 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:11.622392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:11.623034 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:11.623052 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:11.623491 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:11.623494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:11.623576 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:11.624050 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:11.624062 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:11.625818 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":107,"threads_started":1}
W20260809 07:54:11.626108 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:11.620664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:37155 (local address 127.28.112.62:38715)
0809 07:54:11.620834 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:11.620838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:11.620988 (+ 150us) server_negotiation.cc:407] Connection header received
0809 07:54:11.621110 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:11.621114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:11.621158 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:11.621225 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:11.621710 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:11.622282 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:11.623172 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:11.623381 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:11.623470 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:11.626004 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.1:37155: BlockingRecv error: recv got EOF from 127.28.112.1:37155 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 09 07:54:11 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 07:54:11 dist-test-slave-vpvm krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.112.2@KRBTEST.COM for oryx/127.28.112.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 07:54:11.802394 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:11.797113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38715 (local address 127.28.112.2:38559)
0809 07:54:11.797392 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:11.797405 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:11.797510 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:11.797753 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:11.797756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:11.797980 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:11.798087 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:11.798093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:11.798839 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:11.798843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:11.799410 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:11.799418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:11.799852 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:11.799856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:11.799933 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:11.800431 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:11.800446 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:11.802267 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.112.62:38715: Server oryx/127.28.112.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":86,"threads_started":1}
W20260809 07:54:11.802548 10517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:11.797196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:38559 (local address 127.28.112.62:38715)
0809 07:54:11.797342 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:11.797347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:11.797468 (+ 121us) server_negotiation.cc:407] Connection header received
0809 07:54:11.797593 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:11.797597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:11.797645 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:11.797712 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:11.798210 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:11.798731 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:11.799522 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:11.799750 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:11.799851 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:11.802405 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.112.2:38559: BlockingRecv error: recv got EOF from 127.28.112.2:38559 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 07:54:12.065361 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 9900
I20260809 07:54:12.090171 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 10122
I20260809 07:54:12.098834 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 10318
I20260809 07:54:12.109377 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 9791
2026-08-09T07:54:12Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18200 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-09T07:54:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:54:12Z Disabled control of system clock
I20260809 07:54:12.146294 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:44391
--webserver_interface=127.28.112.62
--webserver_port=0
--builtin_ntp_servers=127.28.112.20:35105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:44391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 07:54:12.256825 10589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:54:12.257126 10589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:54:12.257176 10589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:54:12.261178 10589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:54:12.261235 10589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:54:12.261257 10589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:54:12.261278 10589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:54:12.261307 10589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:54:12.266065 10589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:35105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:44391
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:44391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:54:12.267102 10589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:54:12.268450 10589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:54:12.276635 10598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:12.276643 10602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:12.276646 10600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:12.276783 10589 server_base.cc:1034] running on GCE node
I20260809 07:54:12.277448 10589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:54:12.278508 10589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:54:12.279663 10589 hybrid_clock.cc:648] HybridClock initialized: now 1786262052279649 us; error 25 us; skew 500 ppm
I20260809 07:54:12.281374 10589 webserver.cc:460] Webserver started at http://127.28.112.62:42431/ using document root <none> and password file <none>
I20260809 07:54:12.281900 10589 fs_manager.cc:359] Metadata directory not provided
I20260809 07:54:12.282004 10589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:54:12.282212 10589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:54:12.284166 10589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "512d73350f5041b19da70c8ed9dba46a"
format_stamp: "Formatted at 2026-08-09 07:54:12 on dist-test-slave-vpvm"
server_key: "614e678eb6968512e9ab054ce27469cd"
server_key_iv: "0e2da92440fc16358de242c73c3ca440"
server_key_version: "encryptionkey@0"
I20260809 07:54:12.284684 10589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "512d73350f5041b19da70c8ed9dba46a"
format_stamp: "Formatted at 2026-08-09 07:54:12 on dist-test-slave-vpvm"
server_key: "614e678eb6968512e9ab054ce27469cd"
server_key_iv: "0e2da92440fc16358de242c73c3ca440"
server_key_version: "encryptionkey@0"
I20260809 07:54:12.288193 10589 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 07:54:12.290504 10611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:12.291659 10589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:54:12.291782 10589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "512d73350f5041b19da70c8ed9dba46a"
format_stamp: "Formatted at 2026-08-09 07:54:12 on dist-test-slave-vpvm"
server_key: "614e678eb6968512e9ab054ce27469cd"
server_key_iv: "0e2da92440fc16358de242c73c3ca440"
server_key_version: "encryptionkey@0"
I20260809 07:54:12.291877 10589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:54:12.317196 10589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:54:12.317866 10589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:54:12.318068 10589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:54:12.324707 10589 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:44391
I20260809 07:54:12.324725 10693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:44391 every 8 connection(s)
I20260809 07:54:12.325623 10589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:54:12.328904 10696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:54:12.331537 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 10589
I20260809 07:54:12.331636 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:54:12.331916 29120 external_mini_cluster.cc:1428] Setting key 4b644da49cbcaf38c3812f66c85e43e7
I20260809 07:54:12.334692 10696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a: Bootstrap starting.
I20260809 07:54:12.337088 10696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a: Neither blocks nor log segments found. Creating new log.
I20260809 07:54:12.337944 10696 log.cc:824] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a: Log is configured to *not* fsync() on all Append() calls
I20260809 07:54:12.340194 10696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a: No bootstrap required, opened a new log
I20260809 07:54:12.341210 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:12.333314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53478 (local address 127.28.112.62:44391)
0809 07:54:12.333908 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:12.333924 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:12.333992 (+ 68us) server_negotiation.cc:407] Connection header received
0809 07:54:12.334591 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:12.334607 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:12.335011 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:12.335279 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:12.335615 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:12.336546 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:12.337744 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:12.338281 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:12.338455 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:12.339206 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:54:12.339213 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:54:12.339228 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:54:12.339708 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:54:12.339777 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:54:12.339803 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:54:12.339902 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:12.340047 (+ 145us) server_negotiation.cc:299] Negotiation successful
0809 07:54:12.340309 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":114,"threads_started":1}
I20260809 07:54:12.343917 10696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512d73350f5041b19da70c8ed9dba46a" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 44391 } }
I20260809 07:54:12.344153 10696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:54:12.344210 10696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 512d73350f5041b19da70c8ed9dba46a, State: Initialized, Role: FOLLOWER
I20260809 07:54:12.344602 10696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512d73350f5041b19da70c8ed9dba46a" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 44391 } }
I20260809 07:54:12.344722 10696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:54:12.344771 10696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:54:12.344842 10696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:54:12.345678 10696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512d73350f5041b19da70c8ed9dba46a" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 44391 } }
I20260809 07:54:12.346026 10696 leader_election.cc:302] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 512d73350f5041b19da70c8ed9dba46a; no voters:
I20260809 07:54:12.346315 10696 leader_election.cc:288] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:54:12.346416 10705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:54:12.346635 10705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a [term 1 LEADER]: Becoming Leader. State: Replica: 512d73350f5041b19da70c8ed9dba46a, State: Running, Role: LEADER
I20260809 07:54:12.346974 10705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512d73350f5041b19da70c8ed9dba46a" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 44391 } }
I20260809 07:54:12.347479 10696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:54:12.348465 10708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 512d73350f5041b19da70c8ed9dba46a. Latest consensus state: current_term: 1 leader_uuid: "512d73350f5041b19da70c8ed9dba46a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512d73350f5041b19da70c8ed9dba46a" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 44391 } } }
I20260809 07:54:12.348596 10708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a [sys.catalog]: This master's current role is: LEADER
I20260809 07:54:12.348971 10723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:54:12.348950 10706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "512d73350f5041b19da70c8ed9dba46a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512d73350f5041b19da70c8ed9dba46a" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 44391 } } }
I20260809 07:54:12.349066 10706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a [sys.catalog]: This master's current role is: LEADER
I20260809 07:54:12.352233 10723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:54:12.357887 10723 catalog_manager.cc:1347] Generated new cluster ID: 585b619da62e405caa89824f58934afa
I20260809 07:54:12.357980 10723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:54:12.374452 10723 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:54:12.375461 10723 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:54:12.381009 10723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 512d73350f5041b19da70c8ed9dba46a: Generated new TSK 0
I20260809 07:54:12.381592 10723 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:54:12.388083 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:44391
--builtin_ntp_servers=127.28.112.20:35105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260809 07:54:12.497540 10738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:54:12.497795 10738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:54:12.497829 10738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:54:12.501770 10738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:54:12.501820 10738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:54:12.501873 10738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:54:12.506485 10738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:35105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:44391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:54:12.507517 10738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:54:12.508857 10738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:54:12.518118 10748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:12.518163 10738 server_base.cc:1034] running on GCE node
W20260809 07:54:12.518127 10745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:12.518127 10744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:12.518708 10738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:54:12.519290 10738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:54:12.520444 10738 hybrid_clock.cc:648] HybridClock initialized: now 1786262052520427 us; error 31 us; skew 500 ppm
I20260809 07:54:12.522241 10738 webserver.cc:460] Webserver started at http://127.28.112.1:43103/ using document root <none> and password file <none>
I20260809 07:54:12.522809 10738 fs_manager.cc:359] Metadata directory not provided
I20260809 07:54:12.522884 10738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:54:12.523087 10738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:54:12.525193 10738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "8fd61d1a8dfc45e08c3ff71375ffba4f"
format_stamp: "Formatted at 2026-08-09 07:54:12 on dist-test-slave-vpvm"
server_key: "d72aaba0cab9ee70ada1b79fe081ed87"
server_key_iv: "1ada61ce1b81414366a4abdcd5f17922"
server_key_version: "encryptionkey@0"
I20260809 07:54:12.525755 10738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "8fd61d1a8dfc45e08c3ff71375ffba4f"
format_stamp: "Formatted at 2026-08-09 07:54:12 on dist-test-slave-vpvm"
server_key: "d72aaba0cab9ee70ada1b79fe081ed87"
server_key_iv: "1ada61ce1b81414366a4abdcd5f17922"
server_key_version: "encryptionkey@0"
I20260809 07:54:12.529408 10738 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 07:54:12.531876 10757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:12.533044 10738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:54:12.533164 10738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "8fd61d1a8dfc45e08c3ff71375ffba4f"
format_stamp: "Formatted at 2026-08-09 07:54:12 on dist-test-slave-vpvm"
server_key: "d72aaba0cab9ee70ada1b79fe081ed87"
server_key_iv: "1ada61ce1b81414366a4abdcd5f17922"
server_key_version: "encryptionkey@0"
I20260809 07:54:12.533259 10738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:54:12.557945 10738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:54:12.558626 10738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:54:12.558797 10738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:54:12.559372 10738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:54:12.560321 10738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:54:12.560369 10738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:12.560424 10738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:54:12.560452 10738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:12.568660 10738 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:41891
I20260809 07:54:12.568701 10929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:41891 every 8 connection(s)
I20260809 07:54:12.569613 10738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:54:12.574963 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 10738
I20260809 07:54:12.575068 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:54:12.575340 29120 external_mini_cluster.cc:1428] Setting key fd00818ae093c45a878b9db5caabc7ad
I20260809 07:54:12.577323 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:44391
--builtin_ntp_servers=127.28.112.20:35105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 07:54:12.578682 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:12.571345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:56445 (local address 127.28.112.62:44391)
0809 07:54:12.571464 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:12.571468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:12.572456 (+ 988us) server_negotiation.cc:407] Connection header received
0809 07:54:12.573831 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:12.573835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:12.573895 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:12.574000 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:12.574877 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:12.575473 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:12.576734 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:12.577040 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:12.577167 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:12.578230 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:54:12.578234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:54:12.578237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:54:12.578252 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:54:12.578274 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:54:12.578284 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:54:12.578359 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:12.578484 (+ 125us) server_negotiation.cc:299] Negotiation successful
0809 07:54:12.578541 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 07:54:12.579624 10934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:12.571680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:44391 (local address 127.28.112.1:56445)
0809 07:54:12.572311 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:12.572349 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:12.573618 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:12.574160 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:12.574167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:12.574248 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:12.574716 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:12.574751 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:12.575594 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:12.575597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:12.576586 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:12.576598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:12.577199 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:12.577201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:12.577342 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:12.577939 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:54:12.577940 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:12.578001 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:54:12.578069 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:54:12.578357 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:54:12.578361 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:54:12.578365 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 07:54:12.578470 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 07:54:12.578678 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":122,"threads_started":1}
I20260809 07:54:12.580744 10930 heartbeater.cc:347] Connected to a master server at 127.28.112.62:44391
I20260809 07:54:12.580971 10930 heartbeater.cc:464] Registering TS with master...
I20260809 07:54:12.581524 10930 heartbeater.cc:511] Master 127.28.112.62:44391 requested a full tablet report, sending...
I20260809 07:54:12.583103 10639 ts_manager.cc:194] Registered new tserver with Master: 8fd61d1a8dfc45e08c3ff71375ffba4f (127.28.112.1:41891)
I20260809 07:54:12.584221 10639 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.112.1:56445
W20260809 07:54:12.686252 10936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:54:12.686471 10936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:54:12.686506 10936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:54:12.690421 10936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:54:12.690469 10936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:54:12.690531 10936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:54:12.695144 10936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:35105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:44391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:54:12.696412 10936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:54:12.697942 10936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:54:12.706950 10943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:12.706951 10946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:12.707011 10949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:12.707119 10936 server_base.cc:1034] running on GCE node
I20260809 07:54:12.707573 10936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:54:12.708138 10936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:54:12.709304 10936 hybrid_clock.cc:648] HybridClock initialized: now 1786262052709286 us; error 32 us; skew 500 ppm
I20260809 07:54:12.711023 10936 webserver.cc:460] Webserver started at http://127.28.112.2:36693/ using document root <none> and password file <none>
I20260809 07:54:12.711544 10936 fs_manager.cc:359] Metadata directory not provided
I20260809 07:54:12.711609 10936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:54:12.711803 10936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:54:12.713696 10936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "6786bb3e1a8d40abba44d1be0321293e"
format_stamp: "Formatted at 2026-08-09 07:54:12 on dist-test-slave-vpvm"
server_key: "b456e92ec5cc594806c44c306dd9b9e5"
server_key_iv: "c41b2ac6bcaf502325607db15f19cf17"
server_key_version: "encryptionkey@0"
I20260809 07:54:12.714221 10936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "6786bb3e1a8d40abba44d1be0321293e"
format_stamp: "Formatted at 2026-08-09 07:54:12 on dist-test-slave-vpvm"
server_key: "b456e92ec5cc594806c44c306dd9b9e5"
server_key_iv: "c41b2ac6bcaf502325607db15f19cf17"
server_key_version: "encryptionkey@0"
I20260809 07:54:12.717528 10936 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260809 07:54:12.719796 10957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:12.720898 10936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:54:12.721009 10936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "6786bb3e1a8d40abba44d1be0321293e"
format_stamp: "Formatted at 2026-08-09 07:54:12 on dist-test-slave-vpvm"
server_key: "b456e92ec5cc594806c44c306dd9b9e5"
server_key_iv: "c41b2ac6bcaf502325607db15f19cf17"
server_key_version: "encryptionkey@0"
I20260809 07:54:12.721103 10936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:54:12.736730 10936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:54:12.737277 10936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:54:12.737432 10936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:54:12.737937 10936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:54:12.738842 10936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:54:12.738888 10936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:12.738937 10936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:54:12.738967 10936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:12.747002 10936 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:46493
I20260809 07:54:12.747061 11126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:46493 every 8 connection(s)
I20260809 07:54:12.747812 10936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:54:12.754171 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 10936
I20260809 07:54:12.754282 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:54:12.754595 29120 external_mini_cluster.cc:1428] Setting key 9e7cc304efe673622cee661a47f393cf
I20260809 07:54:12.756307 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:12.749478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:39043 (local address 127.28.112.62:44391)
0809 07:54:12.749603 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:12.749607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:12.750601 (+ 994us) server_negotiation.cc:407] Connection header received
0809 07:54:12.751868 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:12.751871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:12.751915 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:12.751979 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:12.752748 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:12.753242 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:12.754572 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:12.754816 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:12.754898 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:12.755875 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:54:12.755879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:54:12.755881 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:54:12.755894 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:54:12.755914 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:54:12.755922 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:54:12.755979 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:12.756115 (+ 136us) server_negotiation.cc:299] Negotiation successful
0809 07:54:12.756173 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 07:54:12.756537 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:44391
--builtin_ntp_servers=127.28.112.20:35105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 07:54:12.757112 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:12.749825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:44391 (local address 127.28.112.2:39043)
0809 07:54:12.750469 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:12.750506 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:12.751684 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:12.752117 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:12.752123 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:12.752210 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:12.752644 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:12.752656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:12.753325 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:12.753327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:12.754455 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:12.754468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:12.754969 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:12.754973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:12.755110 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:12.755610 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:54:12.755612 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:12.755661 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:54:12.755736 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:54:12.755989 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:54:12.755998 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:54:12.756004 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 07:54:12.756107 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 07:54:12.756369 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":143,"threads_started":1}
I20260809 07:54:12.758749 11127 heartbeater.cc:347] Connected to a master server at 127.28.112.62:44391
I20260809 07:54:12.758946 11127 heartbeater.cc:464] Registering TS with master...
I20260809 07:54:12.759429 11127 heartbeater.cc:511] Master 127.28.112.62:44391 requested a full tablet report, sending...
I20260809 07:54:12.760439 10639 ts_manager.cc:194] Registered new tserver with Master: 6786bb3e1a8d40abba44d1be0321293e (127.28.112.2:46493)
I20260809 07:54:12.760963 10639 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.112.2:39043
W20260809 07:54:12.866063 11130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:54:12.866279 11130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:54:12.866314 11130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:54:12.870227 11130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:54:12.870275 11130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:54:12.870337 11130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:54:12.875007 11130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:35105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:44391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:54:12.875964 11130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:54:12.877225 11130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:54:12.886348 11143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:12.886356 11140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:12.886488 11139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:12.886632 11130 server_base.cc:1034] running on GCE node
I20260809 07:54:12.887171 11130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:54:12.887765 11130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:54:12.888922 11130 hybrid_clock.cc:648] HybridClock initialized: now 1786262052888904 us; error 31 us; skew 500 ppm
I20260809 07:54:12.890729 11130 webserver.cc:460] Webserver started at http://127.28.112.3:46813/ using document root <none> and password file <none>
I20260809 07:54:12.891278 11130 fs_manager.cc:359] Metadata directory not provided
I20260809 07:54:12.891345 11130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:54:12.891557 11130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:54:12.893517 11130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "040b2857fb144f108757b143e1efdc21"
format_stamp: "Formatted at 2026-08-09 07:54:12 on dist-test-slave-vpvm"
server_key: "db9e6c2c532804820010ae466e22283d"
server_key_iv: "c64e65e30217743f92091dc58b46ec68"
server_key_version: "encryptionkey@0"
I20260809 07:54:12.894119 11130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "040b2857fb144f108757b143e1efdc21"
format_stamp: "Formatted at 2026-08-09 07:54:12 on dist-test-slave-vpvm"
server_key: "db9e6c2c532804820010ae466e22283d"
server_key_iv: "c64e65e30217743f92091dc58b46ec68"
server_key_version: "encryptionkey@0"
I20260809 07:54:12.897614 11130 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 07:54:12.900152 11153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:12.901320 11130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:54:12.901439 11130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "040b2857fb144f108757b143e1efdc21"
format_stamp: "Formatted at 2026-08-09 07:54:12 on dist-test-slave-vpvm"
server_key: "db9e6c2c532804820010ae466e22283d"
server_key_iv: "c64e65e30217743f92091dc58b46ec68"
server_key_version: "encryptionkey@0"
I20260809 07:54:12.901531 11130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:54:12.922228 11130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:54:12.922886 11130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:54:12.923048 11130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:54:12.923640 11130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:54:12.924588 11130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:54:12.924634 11130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:12.924687 11130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:54:12.924716 11130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:12.932968 11130 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:46803
I20260809 07:54:12.932993 11314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:46803 every 8 connection(s)
I20260809 07:54:12.933802 11130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:54:12.942307 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:12.935507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:44511 (local address 127.28.112.62:44391)
0809 07:54:12.935671 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:12.935676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:12.936693 (+ 1017us) server_negotiation.cc:407] Connection header received
0809 07:54:12.938125 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:12.938128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:12.938177 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:12.938251 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:12.939105 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:12.939645 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:12.940672 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:12.940881 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:12.940959 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:12.941848 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:54:12.941851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:54:12.941853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:54:12.941866 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:54:12.941884 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:54:12.941892 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:54:12.941975 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:12.942137 (+ 162us) server_negotiation.cc:299] Negotiation successful
0809 07:54:12.942193 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 07:54:12.942992 11319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:12.935899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:44391 (local address 127.28.112.3:44511)
0809 07:54:12.936551 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:12.936589 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:12.937892 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:12.938395 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:12.938401 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:12.938497 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:12.938965 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:12.938978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:12.939738 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:12.939740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:12.940577 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:12.940587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:12.940994 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:12.940996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:12.941128 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:12.941613 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:54:12.941614 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:12.941658 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:54:12.941716 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:54:12.941986 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:54:12.941993 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:54:12.941997 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 07:54:12.942086 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 07:54:12.942291 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":103,"threads_started":1}
I20260809 07:54:12.943500 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 11130
I20260809 07:54:12.943602 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:54:12.943924 29120 external_mini_cluster.cc:1428] Setting key f1b4460679022ea82a3a846c44080217
I20260809 07:54:12.944177 11316 heartbeater.cc:347] Connected to a master server at 127.28.112.62:44391
I20260809 07:54:12.944409 11316 heartbeater.cc:464] Registering TS with master...
I20260809 07:54:12.944965 11316 heartbeater.cc:511] Master 127.28.112.62:44391 requested a full tablet report, sending...
I20260809 07:54:12.945945 10639 ts_manager.cc:194] Registered new tserver with Master: 040b2857fb144f108757b143e1efdc21 (127.28.112.3:46803)
I20260809 07:54:12.946511 10639 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.112.3:44511
I20260809 07:54:12.957520 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 07:54:12.960193 10703 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:12.959289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53492 (local address 127.28.112.62:44391)
0809 07:54:12.959421 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:12.959425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:12.959598 (+ 173us) server_negotiation.cc:407] Connection header received
0809 07:54:12.959711 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:12.959714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:12.959756 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:12.959817 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:12.960075 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53492: BlockingRecv error: recv got EOF from 127.0.0.1:53492 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 07:54:12.960325 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 10738
I20260809 07:54:12.973866 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 10936
I20260809 07:54:12.979161 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 11130
I20260809 07:54:12.999511 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 10589
2026-08-09T07:54:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (885 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-09T07:54:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:54:13Z Disabled control of system clock
I20260809 07:54:13.031878 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:39999
--webserver_interface=127.28.112.62
--webserver_port=0
--builtin_ntp_servers=127.28.112.20:33977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:39999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 07:54:13.140602 11340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:54:13.140828 11340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:54:13.140861 11340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:54:13.144707 11340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:54:13.144757 11340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:54:13.144771 11340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:54:13.144786 11340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:54:13.144799 11340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:54:13.149278 11340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:33977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:39999
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:39999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:54:13.150171 11340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:54:13.151410 11340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:54:13.159490 11350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:13.159459 11347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:13.159539 11340 server_base.cc:1034] running on GCE node
W20260809 07:54:13.159596 11348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:13.160223 11340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:54:13.161177 11340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:54:13.162326 11340 hybrid_clock.cc:648] HybridClock initialized: now 1786262053162310 us; error 26 us; skew 500 ppm
I20260809 07:54:13.163975 11340 webserver.cc:460] Webserver started at http://127.28.112.62:40695/ using document root <none> and password file <none>
I20260809 07:54:13.164481 11340 fs_manager.cc:359] Metadata directory not provided
I20260809 07:54:13.164537 11340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:54:13.164713 11340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:54:13.166684 11340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "494ce904611e4b7f92ccb8a42f12da37"
format_stamp: "Formatted at 2026-08-09 07:54:13 on dist-test-slave-vpvm"
server_key: "eb0e57480d94ffc83e9d78ef6eb82eb6"
server_key_iv: "0d31f70c927df076cb23306cefbb9395"
server_key_version: "encryptionkey@0"
I20260809 07:54:13.167191 11340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "494ce904611e4b7f92ccb8a42f12da37"
format_stamp: "Formatted at 2026-08-09 07:54:13 on dist-test-slave-vpvm"
server_key: "eb0e57480d94ffc83e9d78ef6eb82eb6"
server_key_iv: "0d31f70c927df076cb23306cefbb9395"
server_key_version: "encryptionkey@0"
I20260809 07:54:13.170596 11340 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 07:54:13.172796 11362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:13.174016 11340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:54:13.174142 11340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "494ce904611e4b7f92ccb8a42f12da37"
format_stamp: "Formatted at 2026-08-09 07:54:13 on dist-test-slave-vpvm"
server_key: "eb0e57480d94ffc83e9d78ef6eb82eb6"
server_key_iv: "0d31f70c927df076cb23306cefbb9395"
server_key_version: "encryptionkey@0"
I20260809 07:54:13.174233 11340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:54:13.178530 11340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:54:13.179075 11340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:54:13.179219 11340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:54:13.185348 11340 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:39999
I20260809 07:54:13.185354 11450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:39999 every 8 connection(s)
I20260809 07:54:13.186241 11340 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:54:13.186862 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 11340
I20260809 07:54:13.186977 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:54:13.187270 29120 external_mini_cluster.cc:1428] Setting key c1247d6227bed5e214b752c54492049c
I20260809 07:54:13.189436 11451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:54:13.193426 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:13.188587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.28.112.62:39999)
0809 07:54:13.189588 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:13.189608 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:13.189664 (+ 56us) server_negotiation.cc:407] Connection header received
0809 07:54:13.190503 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:13.190525 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:13.190973 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:13.191323 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:13.192000 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:54:13.192006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:54:13.192020 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:54:13.192030 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:54:13.192087 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:54:13.192108 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:54:13.192252 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:13.192321 (+ 69us) server_negotiation.cc:299] Negotiation successful
0809 07:54:13.192560 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":464,"threads_started":1}
I20260809 07:54:13.195993 11451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37: Bootstrap starting.
I20260809 07:54:13.198009 11451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37: Neither blocks nor log segments found. Creating new log.
I20260809 07:54:13.198673 11451 log.cc:824] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37: Log is configured to *not* fsync() on all Append() calls
I20260809 07:54:13.200469 11451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37: No bootstrap required, opened a new log
I20260809 07:54:13.203843 11451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494ce904611e4b7f92ccb8a42f12da37" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39999 } }
I20260809 07:54:13.204051 11451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:54:13.204099 11451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 494ce904611e4b7f92ccb8a42f12da37, State: Initialized, Role: FOLLOWER
I20260809 07:54:13.204494 11451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494ce904611e4b7f92ccb8a42f12da37" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39999 } }
I20260809 07:54:13.204619 11451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:54:13.204679 11451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:54:13.204771 11451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:54:13.205570 11451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494ce904611e4b7f92ccb8a42f12da37" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39999 } }
I20260809 07:54:13.205885 11451 leader_election.cc:302] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 494ce904611e4b7f92ccb8a42f12da37; no voters:
I20260809 07:54:13.206228 11451 leader_election.cc:288] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:54:13.206341 11460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:54:13.206593 11460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37 [term 1 LEADER]: Becoming Leader. State: Replica: 494ce904611e4b7f92ccb8a42f12da37, State: Running, Role: LEADER
I20260809 07:54:13.206928 11460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494ce904611e4b7f92ccb8a42f12da37" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39999 } }
I20260809 07:54:13.207396 11451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:54:13.208458 11462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 494ce904611e4b7f92ccb8a42f12da37. Latest consensus state: current_term: 1 leader_uuid: "494ce904611e4b7f92ccb8a42f12da37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494ce904611e4b7f92ccb8a42f12da37" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39999 } } }
I20260809 07:54:13.208590 11462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37 [sys.catalog]: This master's current role is: LEADER
I20260809 07:54:13.208968 11475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:54:13.208935 11461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "494ce904611e4b7f92ccb8a42f12da37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494ce904611e4b7f92ccb8a42f12da37" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39999 } } }
I20260809 07:54:13.209565 11461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37 [sys.catalog]: This master's current role is: LEADER
I20260809 07:54:13.212080 11475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:54:13.217556 11475 catalog_manager.cc:1347] Generated new cluster ID: 2b88dc6d429f4210b16b1f2cbba82eab
I20260809 07:54:13.217624 11475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:54:13.264333 11475 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:54:13.264724 11475 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:54:13.273353 11475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 494ce904611e4b7f92ccb8a42f12da37: Generated new TSK 0
I20260809 07:54:13.274051 11475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:54:13.288244 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:39999
--builtin_ntp_servers=127.28.112.20:33977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 07:54:13.397307 11489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:54:13.397557 11489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:54:13.397594 11489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:54:13.401510 11489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:54:13.401561 11489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:54:13.401614 11489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:54:13.406272 11489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:33977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:39999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:54:13.407312 11489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:54:13.408751 11489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:54:13.418009 11500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:13.418001 11496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:13.418001 11497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:13.418289 11489 server_base.cc:1034] running on GCE node
I20260809 07:54:13.418682 11489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:54:13.419275 11489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:54:13.420467 11489 hybrid_clock.cc:648] HybridClock initialized: now 1786262053420448 us; error 31 us; skew 500 ppm
I20260809 07:54:13.422294 11489 webserver.cc:460] Webserver started at http://127.28.112.1:45147/ using document root <none> and password file <none>
I20260809 07:54:13.422827 11489 fs_manager.cc:359] Metadata directory not provided
I20260809 07:54:13.422891 11489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:54:13.423084 11489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:54:13.425057 11489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "1d0c7e11879b442f88839b8201fa46bb"
format_stamp: "Formatted at 2026-08-09 07:54:13 on dist-test-slave-vpvm"
server_key: "a39bb05d6526696d9bba0824a564a68f"
server_key_iv: "0623b04303823936d51e6e413e3c7838"
server_key_version: "encryptionkey@0"
I20260809 07:54:13.425570 11489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "1d0c7e11879b442f88839b8201fa46bb"
format_stamp: "Formatted at 2026-08-09 07:54:13 on dist-test-slave-vpvm"
server_key: "a39bb05d6526696d9bba0824a564a68f"
server_key_iv: "0623b04303823936d51e6e413e3c7838"
server_key_version: "encryptionkey@0"
I20260809 07:54:13.429127 11489 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:54:13.431629 11508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:13.432755 11489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:54:13.432862 11489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "1d0c7e11879b442f88839b8201fa46bb"
format_stamp: "Formatted at 2026-08-09 07:54:13 on dist-test-slave-vpvm"
server_key: "a39bb05d6526696d9bba0824a564a68f"
server_key_iv: "0623b04303823936d51e6e413e3c7838"
server_key_version: "encryptionkey@0"
I20260809 07:54:13.432966 11489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:54:13.437237 11489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:54:13.437785 11489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:54:13.437935 11489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:54:13.438469 11489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:54:13.439328 11489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:54:13.439374 11489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:13.439422 11489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:54:13.439450 11489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:13.447786 11489 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:39291
I20260809 07:54:13.447834 11701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:39291 every 8 connection(s)
I20260809 07:54:13.448647 11489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:54:13.454486 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:13.450393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:48345 (local address 127.28.112.62:39999)
0809 07:54:13.450581 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:13.450584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:13.451575 (+ 991us) server_negotiation.cc:407] Connection header received
0809 07:54:13.452894 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:13.452897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:13.452945 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:13.453022 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:13.454058 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:54:13.454062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:54:13.454064 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:54:13.454077 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:54:13.454100 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:54:13.454110 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:54:13.454189 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:13.454308 (+ 119us) server_negotiation.cc:299] Negotiation successful
0809 07:54:13.454354 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 07:54:13.454880 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 11489
I20260809 07:54:13.455003 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:54:13.455277 29120 external_mini_cluster.cc:1428] Setting key 89b19a774f0c4347b190220e8f4e8ca5
I20260809 07:54:13.455381 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:13.450761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:39999 (local address 127.28.112.1:48345)
0809 07:54:13.451426 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:13.451465 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:13.452684 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:13.453180 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:13.453187 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:13.453273 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:13.453789 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:54:13.453791 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:13.453830 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:54:13.453896 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:54:13.454179 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:54:13.454187 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:54:13.454193 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 07:54:13.454343 (+ 150us) client_negotiation.cc:240] Negotiation successful
0809 07:54:13.454602 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":124,"threads_started":1}
I20260809 07:54:13.456624 11703 heartbeater.cc:347] Connected to a master server at 127.28.112.62:39999
I20260809 07:54:13.456833 11703 heartbeater.cc:464] Registering TS with master...
I20260809 07:54:13.457098 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:39999
--builtin_ntp_servers=127.28.112.20:33977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 07:54:13.457228 11703 heartbeater.cc:511] Master 127.28.112.62:39999 requested a full tablet report, sending...
I20260809 07:54:13.458702 11390 ts_manager.cc:194] Registered new tserver with Master: 1d0c7e11879b442f88839b8201fa46bb (127.28.112.1:39291)
W20260809 07:54:13.566737 11708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:54:13.566974 11708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:54:13.567018 11708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:54:13.570904 11708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:54:13.570951 11708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:54:13.571015 11708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:54:13.575942 11708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:33977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:39999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:54:13.576942 11708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:54:13.578362 11708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:54:13.587478 11715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:13.587541 11708 server_base.cc:1034] running on GCE node
W20260809 07:54:13.587610 11720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:13.587494 11718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:13.588140 11708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:54:13.588740 11708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:54:13.589905 11708 hybrid_clock.cc:648] HybridClock initialized: now 1786262053589893 us; error 38 us; skew 500 ppm
I20260809 07:54:13.591624 11708 webserver.cc:460] Webserver started at http://127.28.112.2:34425/ using document root <none> and password file <none>
I20260809 07:54:13.592170 11708 fs_manager.cc:359] Metadata directory not provided
I20260809 07:54:13.592238 11708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:54:13.592442 11708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:54:13.594439 11708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "915978010ab745efb9f9932e41adc1ef"
format_stamp: "Formatted at 2026-08-09 07:54:13 on dist-test-slave-vpvm"
server_key: "1cefd1fefce6e0682bf460db1b63054a"
server_key_iv: "7d53b599410391fe4e665b0c85bad337"
server_key_version: "encryptionkey@0"
I20260809 07:54:13.594959 11708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "915978010ab745efb9f9932e41adc1ef"
format_stamp: "Formatted at 2026-08-09 07:54:13 on dist-test-slave-vpvm"
server_key: "1cefd1fefce6e0682bf460db1b63054a"
server_key_iv: "7d53b599410391fe4e665b0c85bad337"
server_key_version: "encryptionkey@0"
I20260809 07:54:13.598430 11708 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 07:54:13.600785 11732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:13.601871 11708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:54:13.601999 11708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "915978010ab745efb9f9932e41adc1ef"
format_stamp: "Formatted at 2026-08-09 07:54:13 on dist-test-slave-vpvm"
server_key: "1cefd1fefce6e0682bf460db1b63054a"
server_key_iv: "7d53b599410391fe4e665b0c85bad337"
server_key_version: "encryptionkey@0"
I20260809 07:54:13.602097 11708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:54:13.606431 11708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:54:13.607049 11708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:54:13.607200 11708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:54:13.607705 11708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:54:13.608632 11708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:54:13.608680 11708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:13.608728 11708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:54:13.608755 11708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:13.617295 11708 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:43855
I20260809 07:54:13.617326 11905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:43855 every 8 connection(s)
I20260809 07:54:13.618161 11708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:54:13.623835 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:13.619912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:56715 (local address 127.28.112.62:39999)
0809 07:54:13.620092 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:13.620097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:13.621112 (+ 1015us) server_negotiation.cc:407] Connection header received
0809 07:54:13.622368 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:13.622371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:13.622411 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:13.622525 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:13.623492 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:54:13.623495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:54:13.623497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:54:13.623508 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:54:13.623529 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:54:13.623537 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:54:13.623609 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:13.623686 (+ 77us) server_negotiation.cc:299] Negotiation successful
0809 07:54:13.623730 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 07:54:13.624001 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 11708
I20260809 07:54:13.624145 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:54:13.624454 29120 external_mini_cluster.cc:1428] Setting key 36c5fbd4d6ccca4201de4af131492f60
I20260809 07:54:13.624884 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:13.620306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:39999 (local address 127.28.112.2:56715)
0809 07:54:13.620957 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:13.620996 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:13.622175 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:13.622660 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:13.622667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:13.622752 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:13.623257 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:54:13.623258 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:13.623295 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:54:13.623359 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:54:13.623593 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:54:13.623597 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:54:13.623600 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 07:54:13.623722 (+ 122us) client_negotiation.cc:240] Negotiation successful
0809 07:54:13.623960 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":136,"threads_started":1}
I20260809 07:54:13.625988 11906 heartbeater.cc:347] Connected to a master server at 127.28.112.62:39999
I20260809 07:54:13.626219 11906 heartbeater.cc:464] Registering TS with master...
I20260809 07:54:13.626571 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:39999
--builtin_ntp_servers=127.28.112.20:33977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 07:54:13.626665 11906 heartbeater.cc:511] Master 127.28.112.62:39999 requested a full tablet report, sending...
I20260809 07:54:13.627699 11390 ts_manager.cc:194] Registered new tserver with Master: 915978010ab745efb9f9932e41adc1ef (127.28.112.2:43855)
W20260809 07:54:13.742647 11909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:54:13.742877 11909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:54:13.742911 11909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:54:13.746840 11909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:54:13.746887 11909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:54:13.746940 11909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:54:13.751451 11909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:33977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:39999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:54:13.752341 11909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:54:13.753640 11909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:54:13.762820 11923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:13.762962 11909 server_base.cc:1034] running on GCE node
W20260809 07:54:13.762810 11919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:13.762812 11921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:13.763473 11909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:54:13.764076 11909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:54:13.765237 11909 hybrid_clock.cc:648] HybridClock initialized: now 1786262053765218 us; error 30 us; skew 500 ppm
I20260809 07:54:13.766914 11909 webserver.cc:460] Webserver started at http://127.28.112.3:37625/ using document root <none> and password file <none>
I20260809 07:54:13.767462 11909 fs_manager.cc:359] Metadata directory not provided
I20260809 07:54:13.767529 11909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:54:13.767714 11909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:54:13.769661 11909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "be0ec3085c0f4a3a8db5af9802ca36ef"
format_stamp: "Formatted at 2026-08-09 07:54:13 on dist-test-slave-vpvm"
server_key: "9dbcf028553e6d0a642a03320be1e523"
server_key_iv: "08b8733d87829dd9a22d1e7fd24a8507"
server_key_version: "encryptionkey@0"
I20260809 07:54:13.770218 11909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "be0ec3085c0f4a3a8db5af9802ca36ef"
format_stamp: "Formatted at 2026-08-09 07:54:13 on dist-test-slave-vpvm"
server_key: "9dbcf028553e6d0a642a03320be1e523"
server_key_iv: "08b8733d87829dd9a22d1e7fd24a8507"
server_key_version: "encryptionkey@0"
I20260809 07:54:13.773779 11909 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260809 07:54:13.776121 11930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:13.777236 11909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:54:13.777341 11909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "be0ec3085c0f4a3a8db5af9802ca36ef"
format_stamp: "Formatted at 2026-08-09 07:54:13 on dist-test-slave-vpvm"
server_key: "9dbcf028553e6d0a642a03320be1e523"
server_key_iv: "08b8733d87829dd9a22d1e7fd24a8507"
server_key_version: "encryptionkey@0"
I20260809 07:54:13.777431 11909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:54:13.781670 11909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:54:13.782262 11909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:54:13.782413 11909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:54:13.782902 11909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:54:13.783910 11909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:54:13.783993 11909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:13.784075 11909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:54:13.784114 11909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:13.792156 11909 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:44513
I20260809 07:54:13.792188 12108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:44513 every 8 connection(s)
I20260809 07:54:13.792948 11909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:54:13.793200 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 11909
I20260809 07:54:13.793303 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:54:13.793606 29120 external_mini_cluster.cc:1428] Setting key b796da027f1447204e00291821cbcf09
I20260809 07:54:13.798916 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:13.794713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:36587 (local address 127.28.112.62:39999)
0809 07:54:13.794834 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:13.794839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:13.796046 (+ 1207us) server_negotiation.cc:407] Connection header received
0809 07:54:13.797325 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:13.797328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:13.797370 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:13.797441 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:13.798493 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:54:13.798497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:54:13.798500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:54:13.798512 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:54:13.798534 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:54:13.798542 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:54:13.798615 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:13.798743 (+ 128us) server_negotiation.cc:299] Negotiation successful
0809 07:54:13.798787 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 07:54:13.799688 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:13.795184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:39999 (local address 127.28.112.3:36587)
0809 07:54:13.795916 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:13.795966 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:13.797123 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:13.797599 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:13.797606 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:13.797692 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:13.798236 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:54:13.798238 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:13.798276 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:54:13.798344 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:54:13.798611 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:54:13.798619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:54:13.798626 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 07:54:13.798780 (+ 154us) client_negotiation.cc:240] Negotiation successful
0809 07:54:13.799010 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":144,"threads_started":1}
I20260809 07:54:13.800616 12109 heartbeater.cc:347] Connected to a master server at 127.28.112.62:39999
I20260809 07:54:13.800801 12109 heartbeater.cc:464] Registering TS with master...
I20260809 07:54:13.801143 12109 heartbeater.cc:511] Master 127.28.112.62:39999 requested a full tablet report, sending...
I20260809 07:54:13.802042 11390 ts_manager.cc:194] Registered new tserver with Master: be0ec3085c0f4a3a8db5af9802ca36ef (127.28.112.3:44513)
I20260809 07:54:13.808151 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 07:54:13.810870 11456 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:54:13.810079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44438 (local address 127.28.112.62:39999)
0809 07:54:13.810182 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:13.810186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:13.810407 (+ 221us) server_negotiation.cc:407] Connection header received
0809 07:54:13.810523 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:13.810525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:13.810552 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:13.810613 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:13.810783 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44438: BlockingRecv error: recv got EOF from 127.0.0.1:44438 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 07:54:13.811098 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 11489
I20260809 07:54:13.840692 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 11708
I20260809 07:54:13.848666 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 11909
I20260809 07:54:13.867760 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 11340
2026-08-09T07:54:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (868 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:54:13 dist-test-slave-vpvm krb5kdc[12125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:54:13 dist-test-slave-vpvm krb5kdc[12125](info): set up 2 sockets
Aug 09 07:54:13 dist-test-slave-vpvm krb5kdc[12125](info): commencing operation
krb5kdc: starting...
W20260809 07:54:14.999441 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:54:15 dist-test-slave-vpvm krb5kdc[12125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:54:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:54:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:54:15.130669 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:39613
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:39613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:54:15.239297 12145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:54:15.239542 12145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:54:15.239578 12145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:54:15.243438 12145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:54:15.243484 12145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:54:15.243501 12145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:54:15.243516 12145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:54:15.243530 12145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:54:15.248613 12145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:39613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:39613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:54:15.249573 12145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:54:15.250545 12145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:54:15.258601 12154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:15.258658 12160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:15.258810 12145 server_base.cc:1034] running on GCE node
W20260809 07:54:15.258620 12153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:15.259424 12145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:54:15.260468 12145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:54:15.261620 12145 hybrid_clock.cc:648] HybridClock initialized: now 1786262055261604 us; error 26 us; skew 500 ppm
Aug 09 07:54:15 dist-test-slave-vpvm krb5kdc[12125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262055, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:54:15.264201 12145 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:54:15.264990 12145 webserver.cc:460] Webserver started at http://127.28.112.62:40849/ using document root <none> and password file <none>
I20260809 07:54:15.265478 12145 fs_manager.cc:359] Metadata directory not provided
I20260809 07:54:15.265533 12145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:54:15.265695 12145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:54:15.267684 12145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "8dec281b398d46699525075956603deb"
format_stamp: "Formatted at 2026-08-09 07:54:15 on dist-test-slave-vpvm"
server_key: "99d606683f8422ecf8a6aa77cae937a2"
server_key_iv: "943891be9231c27e6f9eaf9dd9aa6499"
server_key_version: "encryptionkey@0"
I20260809 07:54:15.268190 12145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "8dec281b398d46699525075956603deb"
format_stamp: "Formatted at 2026-08-09 07:54:15 on dist-test-slave-vpvm"
server_key: "99d606683f8422ecf8a6aa77cae937a2"
server_key_iv: "943891be9231c27e6f9eaf9dd9aa6499"
server_key_version: "encryptionkey@0"
I20260809 07:54:15.271703 12145 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 07:54:15.274016 12170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:15.275236 12145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:54:15.275334 12145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "8dec281b398d46699525075956603deb"
format_stamp: "Formatted at 2026-08-09 07:54:15 on dist-test-slave-vpvm"
server_key: "99d606683f8422ecf8a6aa77cae937a2"
server_key_iv: "943891be9231c27e6f9eaf9dd9aa6499"
server_key_version: "encryptionkey@0"
I20260809 07:54:15.275410 12145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:54:15.292726 12145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:54:15.295552 12145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:54:15.295696 12145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:54:15.302047 12145 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:39613
I20260809 07:54:15.302289 12249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:39613 every 8 connection(s)
I20260809 07:54:15.302947 12145 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:54:15.305630 12250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:54:15.306396 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 12145
I20260809 07:54:15.306521 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:54:15.306846 29120 external_mini_cluster.cc:1428] Setting key b3fc2c4215ae08c6d28c805de0c31d88
I20260809 07:54:15.311967 12250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb: Bootstrap starting.
I20260809 07:54:15.314654 12250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb: Neither blocks nor log segments found. Creating new log.
Aug 09 07:54:15 dist-test-slave-vpvm krb5kdc[12125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:54:15.315429 12250 log.cc:824] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb: Log is configured to *not* fsync() on all Append() calls
I20260809 07:54:15.317240 12250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb: No bootstrap required, opened a new log
I20260809 07:54:15.320138 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:15.308160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49296 (local address 127.28.112.62:39613)
0809 07:54:15.308557 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:15.308572 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:15.308608 (+ 36us) server_negotiation.cc:407] Connection header received
0809 07:54:15.309231 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:15.309247 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:15.309574 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:15.309835 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:15.310736 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:15.311696 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:15.312805 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:15.313255 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:15.313375 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:15.315944 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:54:15.315949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:54:15.315963 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:54:15.315991 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:54:15.317527 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:54:15.317921 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:54:15.317926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:54:15.317931 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:54:15.318070 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:54:15.318323 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:54:15.318327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:54:15.318329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:54:15.318708 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:54:15.318883 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:15.319175 (+ 292us) server_negotiation.cc:299] Negotiation successful
0809 07:54:15.319450 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":109,"threads_started":1}
I20260809 07:54:15.320761 12250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dec281b398d46699525075956603deb" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39613 } }
I20260809 07:54:15.320974 12250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:54:15.321058 12250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dec281b398d46699525075956603deb, State: Initialized, Role: FOLLOWER
I20260809 07:54:15.321521 12250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dec281b398d46699525075956603deb" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39613 } }
I20260809 07:54:15.321691 12250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:54:15.321753 12250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:54:15.321846 12250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:54:15.322813 12250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dec281b398d46699525075956603deb" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39613 } }
I20260809 07:54:15.323108 12250 leader_election.cc:302] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dec281b398d46699525075956603deb; no voters:
I20260809 07:54:15.323370 12250 leader_election.cc:288] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:54:15.323518 12259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:54:15.323714 12259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb [term 1 LEADER]: Becoming Leader. State: Replica: 8dec281b398d46699525075956603deb, State: Running, Role: LEADER
I20260809 07:54:15.324024 12259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dec281b398d46699525075956603deb" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39613 } }
I20260809 07:54:15.324477 12250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:54:15.325479 12260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dec281b398d46699525075956603deb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dec281b398d46699525075956603deb" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39613 } } }
I20260809 07:54:15.325603 12260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb [sys.catalog]: This master's current role is: LEADER
I20260809 07:54:15.325896 12261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dec281b398d46699525075956603deb. Latest consensus state: current_term: 1 leader_uuid: "8dec281b398d46699525075956603deb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dec281b398d46699525075956603deb" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39613 } } }
I20260809 07:54:15.326042 12273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:54:15.326102 12261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb [sys.catalog]: This master's current role is: LEADER
I20260809 07:54:15.328907 12273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:54:15.334231 12273 catalog_manager.cc:1347] Generated new cluster ID: 7d11d87660694c3f877356586e10f4a7
I20260809 07:54:15.334296 12273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:54:15.349735 12273 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:54:15.351011 12273 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:54:15.362293 12273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb: Generated new TSK 0
I20260809 07:54:15.363055 12273 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
I20260809 07:54:15.389019 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:39613
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:54:15.497444 12293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:54:15.497682 12293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:54:15.497721 12293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:54:15.501603 12293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:54:15.501664 12293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:54:15.501720 12293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:54:15.506875 12293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:39613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:54:15.507861 12293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:54:15.508829 12293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:54:15.517922 12303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:15.517906 12301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:15.517915 12300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:15.518092 12293 server_base.cc:1034] running on GCE node
I20260809 07:54:15.518573 12293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:54:15.519179 12293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:54:15.520334 12293 hybrid_clock.cc:648] HybridClock initialized: now 1786262055520313 us; error 34 us; skew 500 ppm
Aug 09 07:54:15 dist-test-slave-vpvm krb5kdc[12125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262055, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:54:15.522974 12293 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:54:15.523787 12293 webserver.cc:460] Webserver started at http://127.28.112.1:40239/ using document root <none> and password file <none>
I20260809 07:54:15.524303 12293 fs_manager.cc:359] Metadata directory not provided
I20260809 07:54:15.524362 12293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:54:15.524541 12293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:54:15.526633 12293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "613c6dc9eed04791aecb284f3c24a7ec"
format_stamp: "Formatted at 2026-08-09 07:54:15 on dist-test-slave-vpvm"
server_key: "5e2f452ebebf5e8a06812b1f58b2c20e"
server_key_iv: "6a20033954b091dfe547a2cd9a7160ed"
server_key_version: "encryptionkey@0"
I20260809 07:54:15.527158 12293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "613c6dc9eed04791aecb284f3c24a7ec"
format_stamp: "Formatted at 2026-08-09 07:54:15 on dist-test-slave-vpvm"
server_key: "5e2f452ebebf5e8a06812b1f58b2c20e"
server_key_iv: "6a20033954b091dfe547a2cd9a7160ed"
server_key_version: "encryptionkey@0"
I20260809 07:54:15.530678 12293 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 07:54:15.532974 12319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:15.534137 12293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:54:15.534233 12293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "613c6dc9eed04791aecb284f3c24a7ec"
format_stamp: "Formatted at 2026-08-09 07:54:15 on dist-test-slave-vpvm"
server_key: "5e2f452ebebf5e8a06812b1f58b2c20e"
server_key_iv: "6a20033954b091dfe547a2cd9a7160ed"
server_key_version: "encryptionkey@0"
I20260809 07:54:15.534327 12293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:54:15.549448 12293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:54:15.552417 12293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:54:15.552582 12293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:54:15.553154 12293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:54:15.553938 12293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:54:15.554023 12293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:15.554080 12293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:54:15.554108 12293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:15.562428 12293 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:34181
I20260809 07:54:15.562455 12496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:34181 every 8 connection(s)
I20260809 07:54:15.563282 12293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:54:15.564656 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 12293
I20260809 07:54:15.564772 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:54:15.565115 29120 external_mini_cluster.cc:1428] Setting key 74056f04949574a02cab01357298e824
Aug 09 07:54:15 dist-test-slave-vpvm krb5kdc[12125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262055, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.112.2@KRBTEST.COM; defaulting to no policy
I20260809 07:54:15.576858 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:15.565036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:47919 (local address 127.28.112.62:39613)
0809 07:54:15.565159 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:15.565162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:15.566199 (+ 1037us) server_negotiation.cc:407] Connection header received
0809 07:54:15.567664 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:15.567667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:15.567718 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:15.567801 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:15.568909 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:15.569699 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:15.570839 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:15.571062 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:15.571153 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:15.573745 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:54:15.573748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:54:15.573750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:54:15.573773 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:54:15.575092 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:54:15.575552 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:54:15.575556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:54:15.575558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:54:15.575644 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:54:15.575958 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:54:15.575962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:54:15.575964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:54:15.576146 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:54:15.576229 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:15.576640 (+ 411us) server_negotiation.cc:299] Negotiation successful
0809 07:54:15.576740 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 07:54:15.577629 12499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:15.565514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:39613 (local address 127.28.112.1:47919)
0809 07:54:15.566033 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:15.566089 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:15.567449 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:15.567940 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:15.567946 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:15.568308 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:15.568763 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:15.568777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:15.569806 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:15.569808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:15.570712 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:15.570722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:15.571213 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:15.571218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:15.571376 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:15.571900 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:15.571916 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:15.573577 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:54:15.575202 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:54:15.575208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:54:15.575223 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:54:15.575456 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:54:15.575726 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:54:15.575730 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:54:15.575732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:54:15.575860 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:54:15.576233 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:54:15.576238 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:54:15.576454 (+ 216us) client_negotiation.cc:769] Sending connection context
0809 07:54:15.576603 (+ 149us) client_negotiation.cc:240] Negotiation successful
0809 07:54:15.576834 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":139,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.2@KRBTEST.COM" created.
I20260809 07:54:15.578938 12497 heartbeater.cc:347] Connected to a master server at 127.28.112.62:39613
I20260809 07:54:15.579182 12497 heartbeater.cc:464] Registering TS with master...
I20260809 07:54:15.579649 12497 heartbeater.cc:511] Master 127.28.112.62:39613 requested a full tablet report, sending...
I20260809 07:54:15.581131 12197 ts_manager.cc:194] Registered new tserver with Master: 613c6dc9eed04791aecb284f3c24a7ec (127.28.112.1:34181)
I20260809 07:54:15.582353 12197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:47919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
I20260809 07:54:15.592121 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:39613
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:54:15.699577 12503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:54:15.699806 12503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:54:15.699849 12503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:54:15.703743 12503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:54:15.703797 12503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:54:15.703864 12503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:54:15.709745 12503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:39613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:54:15.710726 12503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:54:15.711681 12503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:54:15.720849 12516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:15.720930 12512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:15.720899 12513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:15.721004 12503 server_base.cc:1034] running on GCE node
I20260809 07:54:15.721413 12503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:54:15.722050 12503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:54:15.723214 12503 hybrid_clock.cc:648] HybridClock initialized: now 1786262055723200 us; error 32 us; skew 500 ppm
Aug 09 07:54:15 dist-test-slave-vpvm krb5kdc[12125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262055, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:54:15.725880 12503 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:54:15.726758 12503 webserver.cc:460] Webserver started at http://127.28.112.2:39219/ using document root <none> and password file <none>
I20260809 07:54:15.727306 12503 fs_manager.cc:359] Metadata directory not provided
I20260809 07:54:15.727373 12503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:54:15.727627 12503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:54:15.729552 12503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "00fbecd7fccf4c0a9c72a63350db00fc"
format_stamp: "Formatted at 2026-08-09 07:54:15 on dist-test-slave-vpvm"
server_key: "9bbb666417650bbef8e6b8dc1e9f1ab3"
server_key_iv: "a515a29425240da69e4e0974e479f4e3"
server_key_version: "encryptionkey@0"
I20260809 07:54:15.730113 12503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "00fbecd7fccf4c0a9c72a63350db00fc"
format_stamp: "Formatted at 2026-08-09 07:54:15 on dist-test-slave-vpvm"
server_key: "9bbb666417650bbef8e6b8dc1e9f1ab3"
server_key_iv: "a515a29425240da69e4e0974e479f4e3"
server_key_version: "encryptionkey@0"
I20260809 07:54:15.733529 12503 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 07:54:15.735775 12524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:15.736919 12503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:54:15.737021 12503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "00fbecd7fccf4c0a9c72a63350db00fc"
format_stamp: "Formatted at 2026-08-09 07:54:15 on dist-test-slave-vpvm"
server_key: "9bbb666417650bbef8e6b8dc1e9f1ab3"
server_key_iv: "a515a29425240da69e4e0974e479f4e3"
server_key_version: "encryptionkey@0"
I20260809 07:54:15.737113 12503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:54:15.755981 12503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:54:15.758720 12503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:54:15.758884 12503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:54:15.759467 12503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:54:15.760272 12503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:54:15.760321 12503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:15.760370 12503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:54:15.760401 12503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:15.768417 12503 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:41295
I20260809 07:54:15.768517 12693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:41295 every 8 connection(s)
I20260809 07:54:15.769277 12503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:54:15.777767 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 12503
I20260809 07:54:15.777940 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:54:15.778398 29120 external_mini_cluster.cc:1428] Setting key b1914c4e3d4f2194d2cc92f634b53099
Aug 09 07:54:15 dist-test-slave-vpvm krb5kdc[12125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262055, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:54:15.783016 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:15.771154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:53559 (local address 127.28.112.62:39613)
0809 07:54:15.771280 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:15.771284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:15.772090 (+ 806us) server_negotiation.cc:407] Connection header received
0809 07:54:15.773435 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:15.773438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:15.773497 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:15.773574 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:15.774759 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:15.775286 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:15.776372 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:15.776592 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:15.776680 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:15.779217 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:54:15.779222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:54:15.779225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:54:15.779257 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:54:15.780777 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:54:15.781340 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:54:15.781344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:54:15.781345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:54:15.781441 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:54:15.781835 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:54:15.781840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:54:15.781842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:54:15.782124 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:54:15.782235 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:15.782712 (+ 477us) server_negotiation.cc:299] Negotiation successful
0809 07:54:15.782853 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 07:54:15.783756 12697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:15.771509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:39613 (local address 127.28.112.2:53559)
0809 07:54:15.771951 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:15.771994 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:15.773231 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:15.773746 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:15.773753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:15.774139 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:15.774619 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:15.774632 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:15.775404 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:15.775407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:15.776256 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:15.776265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:15.776728 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:15.776732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:15.776891 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:15.777396 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:15.777412 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:15.779039 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:54:15.780897 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:54:15.780904 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:54:15.780919 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:54:15.781236 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:54:15.781528 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:54:15.781530 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:54:15.781532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:54:15.781704 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:54:15.782229 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:54:15.782235 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:54:15.782481 (+ 246us) client_negotiation.cc:769] Sending connection context
0809 07:54:15.782643 (+ 162us) client_negotiation.cc:240] Negotiation successful
0809 07:54:15.782888 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":114,"threads_started":1}
I20260809 07:54:15.784843 12694 heartbeater.cc:347] Connected to a master server at 127.28.112.62:39613
I20260809 07:54:15.785097 12694 heartbeater.cc:464] Registering TS with master...
I20260809 07:54:15.785584 12694 heartbeater.cc:511] Master 127.28.112.62:39613 requested a full tablet report, sending...
I20260809 07:54:15.786718 12197 ts_manager.cc:194] Registered new tserver with Master: 00fbecd7fccf4c0a9c72a63350db00fc (127.28.112.2:41295)
I20260809 07:54:15.787276 12197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:53559
WARNING: no policy specified for kudu/127.28.112.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
I20260809 07:54:15.806710 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:39613
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:54:15.918025 12702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:54:15.918254 12702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:54:15.918295 12702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:54:15.922179 12702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:54:15.922227 12702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:54:15.922281 12702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:54:15.927443 12702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:39613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:54:15.928392 12702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:54:15.929330 12702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:54:15.938409 12712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:15.938410 12709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:15.938448 12708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:15.938839 12702 server_base.cc:1034] running on GCE node
I20260809 07:54:15.939213 12702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:54:15.939810 12702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:54:15.940963 12702 hybrid_clock.cc:648] HybridClock initialized: now 1786262055940948 us; error 34 us; skew 500 ppm
Aug 09 07:54:15 dist-test-slave-vpvm krb5kdc[12125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262055, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:54:15.943481 12702 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:54:15.944315 12702 webserver.cc:460] Webserver started at http://127.28.112.3:40591/ using document root <none> and password file <none>
I20260809 07:54:15.944860 12702 fs_manager.cc:359] Metadata directory not provided
I20260809 07:54:15.944926 12702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:54:15.945113 12702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:54:15.947116 12702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "7c414c0a117f4a6682f55dae40f748b2"
format_stamp: "Formatted at 2026-08-09 07:54:15 on dist-test-slave-vpvm"
server_key: "05016643cccdd728b3b0397fff714177"
server_key_iv: "fae3c581a4ca50a8ff201918d622ab17"
server_key_version: "encryptionkey@0"
I20260809 07:54:15.947664 12702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "7c414c0a117f4a6682f55dae40f748b2"
format_stamp: "Formatted at 2026-08-09 07:54:15 on dist-test-slave-vpvm"
server_key: "05016643cccdd728b3b0397fff714177"
server_key_iv: "fae3c581a4ca50a8ff201918d622ab17"
server_key_version: "encryptionkey@0"
I20260809 07:54:15.951294 12702 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:54:15.953693 12719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:15.954921 12702 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:54:15.955029 12702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "7c414c0a117f4a6682f55dae40f748b2"
format_stamp: "Formatted at 2026-08-09 07:54:15 on dist-test-slave-vpvm"
server_key: "05016643cccdd728b3b0397fff714177"
server_key_iv: "fae3c581a4ca50a8ff201918d622ab17"
server_key_version: "encryptionkey@0"
I20260809 07:54:15.955120 12702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:54:15.968529 12702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:54:15.971377 12702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:54:15.971544 12702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:54:15.972095 12702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:54:15.972926 12702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:54:15.972972 12702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:15.973024 12702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:54:15.973053 12702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:15.981552 12702 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:33319
I20260809 07:54:15.981585 12899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:33319 every 8 connection(s)
I20260809 07:54:15.982417 12702 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:54:15.982751 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 12702
I20260809 07:54:15.982878 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:54:15.983227 29120 external_mini_cluster.cc:1428] Setting key 2f2b4c69e6e7fd02999a1355d55b6b5d
Aug 09 07:54:15 dist-test-slave-vpvm krb5kdc[12125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262055, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:54:15.995276 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:15.984333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:40779 (local address 127.28.112.62:39613)
0809 07:54:15.984497 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:15.984502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:15.985374 (+ 872us) server_negotiation.cc:407] Connection header received
0809 07:54:15.986782 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:15.986785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:15.986840 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:15.986907 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:15.988033 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:15.988546 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:15.989642 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:15.989851 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:15.989964 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:15.992176 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:54:15.992179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:54:15.992181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:54:15.992204 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:54:15.993406 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:54:15.993869 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:54:15.993872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:54:15.993873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:54:15.993982 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:54:15.994290 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:54:15.994293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:54:15.994295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:54:15.994465 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:54:15.994549 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:15.995003 (+ 454us) server_negotiation.cc:299] Negotiation successful
0809 07:54:15.995118 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 07:54:15.996109 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:15.984749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:39613 (local address 127.28.112.3:40779)
0809 07:54:15.985237 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:15.985291 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:15.986586 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:15.987087 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:15.987094 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:15.987420 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:15.987892 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:15.987905 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:15.988659 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:15.988663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:15.989519 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:15.989528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:15.990004 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:15.990008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:15.990178 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:15.990679 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:15.990695 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:15.992028 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:54:15.993512 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:54:15.993517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:54:15.993531 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:54:15.993777 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:54:15.994072 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:54:15.994076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:54:15.994078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:54:15.994204 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:54:15.994561 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:54:15.994567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:54:15.994781 (+ 214us) client_negotiation.cc:769] Sending connection context
0809 07:54:15.994938 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 07:54:15.995227 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":118,"threads_started":1}
I20260809 07:54:15.997120 12900 heartbeater.cc:347] Connected to a master server at 127.28.112.62:39613
I20260809 07:54:15.997334 12900 heartbeater.cc:464] Registering TS with master...
I20260809 07:54:15.997763 12900 heartbeater.cc:511] Master 127.28.112.62:39613 requested a full tablet report, sending...
I20260809 07:54:15.998904 12195 ts_manager.cc:194] Registered new tserver with Master: 7c414c0a117f4a6682f55dae40f748b2 (127.28.112.3:33319)
I20260809 07:54:15.999619 12195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:40779
I20260809 07:54:16.008333 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 07:54:16.016820 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:16.010057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49304 (local address 127.28.112.62:39613)
0809 07:54:16.010166 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:16.010170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:16.010352 (+ 182us) server_negotiation.cc:407] Connection header received
0809 07:54:16.010496 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:16.010499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:16.010542 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:16.010604 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:16.011243 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.011728 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.012502 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.012717 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.012802 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:16.014030 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:54:16.014034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:54:16.014037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:54:16.014061 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:54:16.015348 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:54:16.015728 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:54:16.015732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:54:16.015734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:54:16.015805 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:54:16.016039 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:54:16.016043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:54:16.016045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:54:16.016240 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:54:16.016355 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:16.016578 (+ 223us) server_negotiation.cc:299] Negotiation successful
0809 07:54:16.016681 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 07:54:16.019407 12195 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:54:16.022467 12195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:54:16.036554 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:16.030582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:41295 (local address 127.0.0.1:48928)
0809 07:54:16.031034 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:16.031082 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:16.031207 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:16.032114 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:16.032121 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:16.032143 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:54:16.032277 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:16.032285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.033815 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.033819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:16.035064 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:16.035075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.036032 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.036035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:16.036146 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:16.036183 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 07:54:16.036284 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 07:54:16.036397 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":91,"threads_started":1}
I20260809 07:54:16.036748 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:16.030698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48928 (local address 127.28.112.2:41295)
0809 07:54:16.031633 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:16.031644 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:16.031664 (+ 20us) server_negotiation.cc:407] Connection header received
0809 07:54:16.031741 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:16.031746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:16.031934 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:16.032405 (+ 471us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:54:16.032579 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.033692 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.035183 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.035903 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.036017 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:16.036175 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:16.036494 (+ 319us) server_negotiation.cc:299] Negotiation successful
0809 07:54:16.036588 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":108,"threads_started":1}
I20260809 07:54:16.036826 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:16.031793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.28.112.1:34181)
0809 07:54:16.032630 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:16.032641 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:16.032663 (+ 22us) server_negotiation.cc:407] Connection header received
0809 07:54:16.032730 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:16.032733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:16.032857 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:16.032979 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:54:16.033180 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.033981 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.035410 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.036184 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.036280 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:16.036404 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:16.036617 (+ 213us) server_negotiation.cc:299] Negotiation successful
0809 07:54:16.036705 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":145,"threads_started":1}
I20260809 07:54:16.037070 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:16.031626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:34181 (local address 127.0.0.1:59802)
0809 07:54:16.032299 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:16.032310 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:16.032385 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:16.032976 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:16.032979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:16.032996 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:54:16.033087 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:16.033093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.034101 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.034104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:16.035289 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:16.035297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.036284 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.036288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:16.036381 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:16.036403 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 07:54:16.036471 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 07:54:16.036543 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":38,"thread_start_us":88,"threads_started":1}
I20260809 07:54:16.036872 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:16.031067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58306 (local address 127.28.112.3:33319)
0809 07:54:16.031460 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:16.031474 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:16.031525 (+ 51us) server_negotiation.cc:407] Connection header received
0809 07:54:16.031723 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:16.031730 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:16.031934 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:16.032087 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:54:16.032497 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.033679 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.035426 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.035945 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.036051 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:16.036189 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:16.036636 (+ 447us) server_negotiation.cc:299] Negotiation successful
0809 07:54:16.036733 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":108,"threads_started":1}
I20260809 07:54:16.036569 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:16.030736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:33319 (local address 127.0.0.1:58306)
0809 07:54:16.031442 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:16.031454 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:16.031552 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:16.032254 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:16.032257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:16.032275 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:54:16.032384 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:16.032391 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.033857 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.033860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:16.035285 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:16.035297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.036248 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.036252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:16.036341 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:16.036361 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 07:54:16.036421 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 07:54:16.036473 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":91,"thread_start_us":124,"threads_started":1}
I20260809 07:54:16.039695 12590 tablet_service.cc:1467] Processing CreateTablet for tablet 5b65ecde4f774bc1bbbc44055fbade2e (DEFAULT_TABLE table=test-table [id=782f25f7e410464b8b8d15e27690bb24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:54:16.039870 12399 tablet_service.cc:1467] Processing CreateTablet for tablet 5b65ecde4f774bc1bbbc44055fbade2e (DEFAULT_TABLE table=test-table [id=782f25f7e410464b8b8d15e27690bb24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:54:16.040779 12590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b65ecde4f774bc1bbbc44055fbade2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:54:16.040870 12399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b65ecde4f774bc1bbbc44055fbade2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:54:16.042670 12800 tablet_service.cc:1467] Processing CreateTablet for tablet 5b65ecde4f774bc1bbbc44055fbade2e (DEFAULT_TABLE table=test-table [id=782f25f7e410464b8b8d15e27690bb24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:54:16.043672 12800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b65ecde4f774bc1bbbc44055fbade2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:54:16.047255 12930 tablet_bootstrap.cc:492] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec: Bootstrap starting.
I20260809 07:54:16.048650 12931 tablet_bootstrap.cc:492] T 5b65ecde4f774bc1bbbc44055fbade2e P 00fbecd7fccf4c0a9c72a63350db00fc: Bootstrap starting.
I20260809 07:54:16.050314 12933 tablet_bootstrap.cc:492] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2: Bootstrap starting.
I20260809 07:54:16.050491 12930 tablet_bootstrap.cc:654] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec: Neither blocks nor log segments found. Creating new log.
I20260809 07:54:16.051229 12931 tablet_bootstrap.cc:654] T 5b65ecde4f774bc1bbbc44055fbade2e P 00fbecd7fccf4c0a9c72a63350db00fc: Neither blocks nor log segments found. Creating new log.
I20260809 07:54:16.052008 12931 log.cc:824] T 5b65ecde4f774bc1bbbc44055fbade2e P 00fbecd7fccf4c0a9c72a63350db00fc: Log is configured to *not* fsync() on all Append() calls
I20260809 07:54:16.052332 12930 log.cc:824] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec: Log is configured to *not* fsync() on all Append() calls
I20260809 07:54:16.052407 12933 tablet_bootstrap.cc:654] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2: Neither blocks nor log segments found. Creating new log.
I20260809 07:54:16.053364 12933 log.cc:824] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2: Log is configured to *not* fsync() on all Append() calls
I20260809 07:54:16.053778 12931 tablet_bootstrap.cc:492] T 5b65ecde4f774bc1bbbc44055fbade2e P 00fbecd7fccf4c0a9c72a63350db00fc: No bootstrap required, opened a new log
I20260809 07:54:16.054044 12931 ts_tablet_manager.cc:1397] T 5b65ecde4f774bc1bbbc44055fbade2e P 00fbecd7fccf4c0a9c72a63350db00fc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 07:54:16.055322 12933 tablet_bootstrap.cc:492] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2: No bootstrap required, opened a new log
I20260809 07:54:16.055424 12930 tablet_bootstrap.cc:492] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec: No bootstrap required, opened a new log
I20260809 07:54:16.055521 12933 ts_tablet_manager.cc:1397] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 07:54:16.055569 12930 ts_tablet_manager.cc:1397] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260809 07:54:16.057181 12931 raft_consensus.cc:348] T 5b65ecde4f774bc1bbbc44055fbade2e P 00fbecd7fccf4c0a9c72a63350db00fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fbecd7fccf4c0a9c72a63350db00fc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41295 } } peers { permanent_uuid: "7c414c0a117f4a6682f55dae40f748b2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 33319 } } peers { permanent_uuid: "613c6dc9eed04791aecb284f3c24a7ec" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 34181 } }
I20260809 07:54:16.057377 12931 raft_consensus.cc:374] T 5b65ecde4f774bc1bbbc44055fbade2e P 00fbecd7fccf4c0a9c72a63350db00fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:54:16.057408 12931 raft_consensus.cc:729] T 5b65ecde4f774bc1bbbc44055fbade2e P 00fbecd7fccf4c0a9c72a63350db00fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fbecd7fccf4c0a9c72a63350db00fc, State: Initialized, Role: FOLLOWER
I20260809 07:54:16.057766 12931 consensus_queue.cc:260] T 5b65ecde4f774bc1bbbc44055fbade2e P 00fbecd7fccf4c0a9c72a63350db00fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fbecd7fccf4c0a9c72a63350db00fc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41295 } } peers { permanent_uuid: "7c414c0a117f4a6682f55dae40f748b2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 33319 } } peers { permanent_uuid: "613c6dc9eed04791aecb284f3c24a7ec" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 34181 } }
I20260809 07:54:16.058393 12694 heartbeater.cc:503] Master 127.28.112.62:39613 was elected leader, sending a full tablet report...
I20260809 07:54:16.058727 12931 ts_tablet_manager.cc:1428] T 5b65ecde4f774bc1bbbc44055fbade2e P 00fbecd7fccf4c0a9c72a63350db00fc: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260809 07:54:16.059134 12933 raft_consensus.cc:348] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fbecd7fccf4c0a9c72a63350db00fc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41295 } } peers { permanent_uuid: "7c414c0a117f4a6682f55dae40f748b2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 33319 } } peers { permanent_uuid: "613c6dc9eed04791aecb284f3c24a7ec" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 34181 } }
I20260809 07:54:16.059360 12933 raft_consensus.cc:374] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:54:16.059432 12933 raft_consensus.cc:729] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c414c0a117f4a6682f55dae40f748b2, State: Initialized, Role: FOLLOWER
I20260809 07:54:16.060043 12933 consensus_queue.cc:260] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fbecd7fccf4c0a9c72a63350db00fc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41295 } } peers { permanent_uuid: "7c414c0a117f4a6682f55dae40f748b2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 33319 } } peers { permanent_uuid: "613c6dc9eed04791aecb284f3c24a7ec" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 34181 } }
I20260809 07:54:16.060307 12930 raft_consensus.cc:348] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fbecd7fccf4c0a9c72a63350db00fc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41295 } } peers { permanent_uuid: "7c414c0a117f4a6682f55dae40f748b2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 33319 } } peers { permanent_uuid: "613c6dc9eed04791aecb284f3c24a7ec" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 34181 } }
I20260809 07:54:16.060593 12930 raft_consensus.cc:374] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:54:16.060681 12930 raft_consensus.cc:729] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 613c6dc9eed04791aecb284f3c24a7ec, State: Initialized, Role: FOLLOWER
I20260809 07:54:16.060752 12900 heartbeater.cc:503] Master 127.28.112.62:39613 was elected leader, sending a full tablet report...
I20260809 07:54:16.061038 12933 ts_tablet_manager.cc:1428] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 07:54:16.061158 12930 consensus_queue.cc:260] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fbecd7fccf4c0a9c72a63350db00fc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41295 } } peers { permanent_uuid: "7c414c0a117f4a6682f55dae40f748b2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 33319 } } peers { permanent_uuid: "613c6dc9eed04791aecb284f3c24a7ec" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 34181 } }
I20260809 07:54:16.061755 12497 heartbeater.cc:503] Master 127.28.112.62:39613 was elected leader, sending a full tablet report...
I20260809 07:54:16.062033 12930 ts_tablet_manager.cc:1428] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260809 07:54:16.064291 12498 tablet.cc:2369] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:54:16.093390 12943 raft_consensus.cc:482] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:54:16.093643 12943 raft_consensus.cc:504] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fbecd7fccf4c0a9c72a63350db00fc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41295 } } peers { permanent_uuid: "7c414c0a117f4a6682f55dae40f748b2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 33319 } } peers { permanent_uuid: "613c6dc9eed04791aecb284f3c24a7ec" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 34181 } }
I20260809 07:54:16.095453 12943 leader_election.cc:288] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fbecd7fccf4c0a9c72a63350db00fc (127.28.112.2:41295), 7c414c0a117f4a6682f55dae40f748b2 (127.28.112.3:33319)
I20260809 07:54:16.098623 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:16.095267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:44171 (local address 127.28.112.2:41295)
0809 07:54:16.095393 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:16.095398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:16.095558 (+ 160us) server_negotiation.cc:407] Connection header received
0809 07:54:16.095765 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:16.095768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:16.095815 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:16.095877 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:54:16.096130 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.096612 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.097704 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.098131 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.098240 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:16.098294 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:16.098447 (+ 153us) server_negotiation.cc:299] Negotiation successful
0809 07:54:16.098511 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 07:54:16.098647 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:16.095164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:41295 (local address 127.28.112.1:44171)
0809 07:54:16.095473 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:16.095486 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:16.095655 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:16.095893 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:16.095896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:16.095911 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:54:16.096019 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:16.096023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.096727 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.096731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:16.097583 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:16.097593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.098240 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.098244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:16.098337 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:16.098357 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 07:54:16.098423 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 07:54:16.098492 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":36,"spinlock_wait_cycles":6528,"thread_start_us":90,"threads_started":1}
I20260809 07:54:16.098783 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:16.095313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:49359 (local address 127.28.112.3:33319)
0809 07:54:16.095435 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:16.095440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:16.095620 (+ 180us) server_negotiation.cc:407] Connection header received
0809 07:54:16.096000 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:16.096002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:16.096043 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:16.096103 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:54:16.096365 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.096843 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.097833 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.098278 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.098384 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:16.098443 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:16.098603 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 07:54:16.098669 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 07:54:16.098805 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:16.095178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:33319 (local address 127.28.112.1:49359)
0809 07:54:16.095536 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:16.095548 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:16.095655 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:16.096141 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:16.096144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:16.096162 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:54:16.096262 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:16.096267 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.096956 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.096959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:16.097724 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:16.097732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.098385 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.098388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:16.098491 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:16.098513 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 07:54:16.098576 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 07:54:16.098641 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":22,"spinlock_wait_cycles":36352,"thread_start_us":124,"threads_started":1}
I20260809 07:54:16.099402 12616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b65ecde4f774bc1bbbc44055fbade2e" candidate_uuid: "613c6dc9eed04791aecb284f3c24a7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fbecd7fccf4c0a9c72a63350db00fc" is_pre_election: true
I20260809 07:54:16.099428 12826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b65ecde4f774bc1bbbc44055fbade2e" candidate_uuid: "613c6dc9eed04791aecb284f3c24a7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c414c0a117f4a6682f55dae40f748b2" is_pre_election: true
I20260809 07:54:16.099676 12616 raft_consensus.cc:2454] T 5b65ecde4f774bc1bbbc44055fbade2e P 00fbecd7fccf4c0a9c72a63350db00fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 613c6dc9eed04791aecb284f3c24a7ec in term 0.
I20260809 07:54:16.099699 12826 raft_consensus.cc:2454] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 613c6dc9eed04791aecb284f3c24a7ec in term 0.
I20260809 07:54:16.100118 12320 leader_election.cc:302] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00fbecd7fccf4c0a9c72a63350db00fc, 613c6dc9eed04791aecb284f3c24a7ec, 7c414c0a117f4a6682f55dae40f748b2; no voters:
I20260809 07:54:16.100379 12943 raft_consensus.cc:2781] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:54:16.100456 12943 raft_consensus.cc:482] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:54:16.100486 12943 raft_consensus.cc:3037] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:54:16.101431 12943 raft_consensus.cc:504] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fbecd7fccf4c0a9c72a63350db00fc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41295 } } peers { permanent_uuid: "7c414c0a117f4a6682f55dae40f748b2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 33319 } } peers { permanent_uuid: "613c6dc9eed04791aecb284f3c24a7ec" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 34181 } }
I20260809 07:54:16.101711 12943 leader_election.cc:288] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [CANDIDATE]: Term 1 election: Requested vote from peers 00fbecd7fccf4c0a9c72a63350db00fc (127.28.112.2:41295), 7c414c0a117f4a6682f55dae40f748b2 (127.28.112.3:33319)
I20260809 07:54:16.102048 12826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b65ecde4f774bc1bbbc44055fbade2e" candidate_uuid: "613c6dc9eed04791aecb284f3c24a7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c414c0a117f4a6682f55dae40f748b2"
I20260809 07:54:16.102068 12616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b65ecde4f774bc1bbbc44055fbade2e" candidate_uuid: "613c6dc9eed04791aecb284f3c24a7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fbecd7fccf4c0a9c72a63350db00fc"
I20260809 07:54:16.102156 12616 raft_consensus.cc:3037] T 5b65ecde4f774bc1bbbc44055fbade2e P 00fbecd7fccf4c0a9c72a63350db00fc [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:54:16.102155 12826 raft_consensus.cc:3037] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:54:16.103113 12616 raft_consensus.cc:2454] T 5b65ecde4f774bc1bbbc44055fbade2e P 00fbecd7fccf4c0a9c72a63350db00fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 613c6dc9eed04791aecb284f3c24a7ec in term 1.
I20260809 07:54:16.103137 12826 raft_consensus.cc:2454] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 613c6dc9eed04791aecb284f3c24a7ec in term 1.
I20260809 07:54:16.103401 12326 leader_election.cc:302] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00fbecd7fccf4c0a9c72a63350db00fc, 613c6dc9eed04791aecb284f3c24a7ec, 7c414c0a117f4a6682f55dae40f748b2; no voters:
I20260809 07:54:16.103583 12943 raft_consensus.cc:2781] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:54:16.103792 12943 raft_consensus.cc:686] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [term 1 LEADER]: Becoming Leader. State: Replica: 613c6dc9eed04791aecb284f3c24a7ec, State: Running, Role: LEADER
I20260809 07:54:16.104111 12943 consensus_queue.cc:237] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fbecd7fccf4c0a9c72a63350db00fc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41295 } } peers { permanent_uuid: "7c414c0a117f4a6682f55dae40f748b2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 33319 } } peers { permanent_uuid: "613c6dc9eed04791aecb284f3c24a7ec" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 34181 } }
I20260809 07:54:16.107016 12195 catalog_manager.cc:5495] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 613c6dc9eed04791aecb284f3c24a7ec (127.28.112.1). New cstate: current_term: 1 leader_uuid: "613c6dc9eed04791aecb284f3c24a7ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fbecd7fccf4c0a9c72a63350db00fc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c414c0a117f4a6682f55dae40f748b2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 33319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "613c6dc9eed04791aecb284f3c24a7ec" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 34181 } health_report { overall_health: HEALTHY } } }
I20260809 07:54:16.117183 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:16.114287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59804 (local address 127.28.112.1:34181)
0809 07:54:16.114385 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:16.114388 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:16.114401 (+ 13us) server_negotiation.cc:407] Connection header received
0809 07:54:16.114529 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:16.114531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:16.114579 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:16.114660 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:54:16.114840 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.115306 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.116090 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:16.116283 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:16.116348 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:16.116557 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:54:16.116935 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:54:16.117017 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:16.117045 (+ 28us) server_negotiation.cc:299] Negotiation successful
0809 07:54:16.117087 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 07:54:16.122984 12616 raft_consensus.cc:1261] T 5b65ecde4f774bc1bbbc44055fbade2e P 00fbecd7fccf4c0a9c72a63350db00fc [term 1 FOLLOWER]: Refusing update from remote peer 613c6dc9eed04791aecb284f3c24a7ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:54:16.122984 12826 raft_consensus.cc:1261] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2 [term 1 FOLLOWER]: Refusing update from remote peer 613c6dc9eed04791aecb284f3c24a7ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:54:16.123767 12948 consensus_queue.cc:1035] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fbecd7fccf4c0a9c72a63350db00fc" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 41295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:54:16.123941 12943 consensus_queue.cc:1035] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c414c0a117f4a6682f55dae40f748b2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 33319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:54:16.132678 12953 mvcc.cc:205] Tried to move back new op lower bound from 7316529381872123904 to 7316529381804843008. Current Snapshot: MvccSnapshot[applied={T|T < 7316529381872123904}]
I20260809 07:54:16.140333 12195 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:54:16.140501 12195 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:54:16.142740 12195 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8dec281b398d46699525075956603deb: Sending DeleteTablet for 3 replicas of tablet 5b65ecde4f774bc1bbbc44055fbade2e
I20260809 07:54:16.143419 12590 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b65ecde4f774bc1bbbc44055fbade2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:54:16 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:48928
I20260809 07:54:16.143541 12800 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b65ecde4f774bc1bbbc44055fbade2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:54:16 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:58306
I20260809 07:54:16.143663 12399 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b65ecde4f774bc1bbbc44055fbade2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:54:16 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:59802
I20260809 07:54:16.143976 12973 tablet_replica.cc:330] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2: stopping tablet replica
I20260809 07:54:16.144093 12974 tablet_replica.cc:330] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec: stopping tablet replica
I20260809 07:54:16.144279 12971 tablet_replica.cc:330] T 5b65ecde4f774bc1bbbc44055fbade2e P 00fbecd7fccf4c0a9c72a63350db00fc: stopping tablet replica
I20260809 07:54:16.144407 12974 raft_consensus.cc:2229] T 5b65ecde4f774bc1bbbc44055fbade2e P 613c6dc9eed04791aecb284f3c24a7ec [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:54:16.144678 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 12293
I20260809 07:54:16.144666 12971 raft_consensus.cc:2229] T 5b65ecde4f774bc1bbbc44055fbade2e P 00fbecd7fccf4c0a9c72a63350db00fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:54:16.145316 12971 raft_consensus.cc:2258] T 5b65ecde4f774bc1bbbc44055fbade2e P 00fbecd7fccf4c0a9c72a63350db00fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:54:16.154908 12971 ts_tablet_manager.cc:1905] T 5b65ecde4f774bc1bbbc44055fbade2e P 00fbecd7fccf4c0a9c72a63350db00fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:54:16.155288 12973 raft_consensus.cc:2229] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260809 07:54:16.155823 12171 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.112.1:34181 (error 108)
I20260809 07:54:16.155882 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 12503
I20260809 07:54:16.159605 12973 raft_consensus.cc:2258] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:54:16.164223 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:16.163906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:34181 (local address 127.0.0.1:59814)
0809 07:54:16.164052 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:16.164127 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:34181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 07:54:16.164253 12973 ts_tablet_manager.cc:1905] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 07:54:16.164479 12171 catalog_manager.cc:4584] TS 613c6dc9eed04791aecb284f3c24a7ec (127.28.112.1:34181): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b65ecde4f774bc1bbbc44055fbade2e: Network error: Client connection negotiation failed: client connection to 127.28.112.1:34181: connect: Connection refused (error 111)
I20260809 07:54:16.166383 12973 ts_tablet_manager.cc:1918] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:54:16.166476 12973 log.cc:1196] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/5b65ecde4f774bc1bbbc44055fbade2e
I20260809 07:54:16.166741 12973 ts_tablet_manager.cc:1939] T 5b65ecde4f774bc1bbbc44055fbade2e P 7c414c0a117f4a6682f55dae40f748b2: Deleting consensus metadata
I20260809 07:54:16.167809 12171 catalog_manager.cc:4841] TS 7c414c0a117f4a6682f55dae40f748b2 (127.28.112.3:33319): tablet 5b65ecde4f774bc1bbbc44055fbade2e (table test-table [id=782f25f7e410464b8b8d15e27690bb24]) successfully deleted
I20260809 07:54:16.167973 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 12702
I20260809 07:54:16.168284 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:16.168043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:41295 (local address 127.0.0.1:48936)
0809 07:54:16.168179 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:16.168222 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.2:41295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260809 07:54:16.177521 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 12145
2026-08-09T07:54:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2311 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:54:16 dist-test-slave-vpvm krb5kdc[12980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:54:16 dist-test-slave-vpvm krb5kdc[12980](info): set up 2 sockets
Aug 09 07:54:16 dist-test-slave-vpvm krb5kdc[12980](info): commencing operation
krb5kdc: starting...
W20260809 07:54:17.330062 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:54:17 dist-test-slave-vpvm krb5kdc[12980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:54:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:54:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.112.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.112.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.112.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/rangeradmin_127.28.112.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.112.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/rangeradmin_127.28.112.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.112.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.112.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.112.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/rangerlookup_127.28.112.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.112.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/rangerlookup_127.28.112.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.112.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.112.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.112.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/HTTP_127.28.112.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.112.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/HTTP_127.28.112.20@KRBTEST.COM.keytab.
I20260809 07:54:17.486696 29120 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task4dUq1L/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/postgres -l logfile start
2026-08-09 07:54:19.707 UTC [13024] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 07:54:19.707 UTC [13024] LOG: listening on IPv4 address "127.28.112.20", port 59317
2026-08-09 07:54:19.711 UTC [13024] LOG: listening on Unix socket "/tmp/.s.PGSQL.59317"
2026-08-09 07:54:19.731 UTC [13028] LOG: database system was shut down at 2026-08-09 07:54:18 UTC
2026-08-09 07:54:19.734 UTC [13024] LOG: database system is ready to accept connections
I20260809 07:54:20.779234 29120 mini_postgres.cc:96] Postgres bound to 59317
2026-08-09 07:54:20.786 UTC [13042] FATAL: database "slave" does not exist
127.28.112.20:59317 - accepting connections
I20260809 07:54:20.786857 29120 mini_ranger.cc:161] Starting Ranger...
I20260809 07:54:20.801079 29120 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 07:54:21.198676 29120 mini_ranger.cc:87] Created ranger Postgres database
I20260809 07:54:21.198760 29120 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ranger-admin
2026-08-09 07:54:21,460 [I] DB FLAVOR :POSTGRES
2026-08-09 07:54:21,461 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 07:54:21,461 [I] Checking connection..
2026-08-09 07:54:21,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 07:54:21,721 [I] Checking connection passed.
2026-08-09 07:54:21,722 [I] --------- Verifying version history table ---------
2026-08-09 07:54:21,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 07:54:21,962 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 07:54:21,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 07:54:22,217 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 07:54:22,217 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 07:54:22,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 07:54:22.469 UTC [13141] WARNING: there is no transaction in progress
2026-08-09 07:54:22,479 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 07:54:22,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 07:54:22,722 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 07:54:22,723 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 07:54:22,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 07:54:22,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 07:54:23,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vpvm.c.gcp-upstream.internal','N') ;"
2026-08-09 07:54:23,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 07:54:23,695 [I] Table x_portal_user does not exist in database ranger
2026-08-09 07:54:23,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 07:54:23,945 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 07:54:23,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 07:54:24,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 07:54:24,451 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 07:54:24,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 07:54:25.352 UTC [13306] WARNING: there is no transaction in progress
2026-08-09 07:54:25.367 UTC [13306] WARNING: there is no transaction in progress
2026-08-09 07:54:25.382 UTC [13306] WARNING: there is no transaction in progress
2026-08-09 07:54:25.396 UTC [13306] WARNING: there is no transaction in progress
2026-08-09 07:54:25.413 UTC [13306] WARNING: there is no transaction in progress
2026-08-09 07:54:25.432 UTC [13306] WARNING: there is no transaction in progress
2026-08-09 07:54:25.494 UTC [13306] WARNING: there is no transaction in progress
2026-08-09 07:54:25.501 UTC [13306] WARNING: there is no transaction in progress
2026-08-09 07:54:25.514 UTC [13306] WARNING: there is no transaction in progress
2026-08-09 07:54:25.523 UTC [13306] WARNING: there is no transaction in progress
2026-08-09 07:54:25.533 UTC [13306] WARNING: there is no transaction in progress
2026-08-09 07:54:25.545 UTC [13306] WARNING: there is no transaction in progress
2026-08-09 07:54:25.554 UTC [13306] WARNING: there is no transaction in progress
2026-08-09 07:54:25.559 UTC [13306] WARNING: there is no transaction in progress
2026-08-09 07:54:25.564 UTC [13306] WARNING: there is no transaction in progress
2026-08-09 07:54:25.570 UTC [13306] WARNING: there is no transaction in progress
2026-08-09 07:54:26,131 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 07:54:26,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 07:54:26,373 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 07:54:26,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 07:54:26,626 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 07:54:26,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 07:54:26,881 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 07:54:26,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 07:54:27,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 07:54:27,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vpvm.c.gcp-upstream.internal';"
2026-08-09 07:54:27,623 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 07:54:27,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 07:54:27,853 [I] DB_PATCHES have already been applied
I20260809 07:54:27.858764 29120 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ranger-admin:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task4dUq1L/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 07:54:27.858829 29120 mini_ranger.cc:193] Using host: 127.28.112.20
I20260809 07:54:27.862928 29120 mini_ranger.cc:227] Ranger admin URL: http://127.28.112.20:56729
Aug 09, 2026 7:54:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 7:54:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 7:54:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 7:54:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 7:54:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 7:54:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 7:54:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 07:54:29.109683 29120 mini_ranger.cc:160] Time spent starting Ranger: real 8.323s user 0.003s sys 0.007s
Aug 09, 2026 7:54:48 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 7:54:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 09, 2026 7:54:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 7:54:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 7:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 7:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 7:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 7:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 7:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 7:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 7:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 7:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 7:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 7:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 7:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 7:54:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 7:54:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 7:54:49 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 07:54:50.302523 29120 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.112.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.112.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.112.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/rangerkms_127.28.112.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.112.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/rangerkms_127.28.112.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.112.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.112.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.112.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/HTTP_127.28.112.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.112.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/HTTP_127.28.112.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 07:54:50 dist-test-slave-vpvm krb5kdc[12980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262090, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 07:54:50.391196 29120 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 07:54:50.401227 29120 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 07:54:51.243584 29120 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 07:54:51.243680 29120 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ranger-kms
I20260809 07:54:51.243762 29120 mini_ranger_kms.cc:227] Using postgres at 127.28.112.20:59317
2026-08-09 07:54:51,473 [I] DB FLAVOR :POSTGRES
2026-08-09 07:54:51,474 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 07:54:51,474 [I] Checking connection
2026-08-09 07:54:51,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 07:54:51,742 [I] connection success
2026-08-09 07:54:51,743 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 07:54:51,743 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 07:54:51,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 07:54:51,980 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 07:54:51,980 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 07:54:51,981 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 07:54:51,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:59317/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 07:54:52,242 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 07:54:52.694921 29120 mini_ranger_kms.cc:310] Created kms service
I20260809 07:54:52.871050 29120 mini_ranger_kms.cc:326] Created kudu user
I20260809 07:54:52.913723 29120 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 07:54:52 dist-test-slave-vpvm krb5kdc[12980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262090, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
I20260809 07:54:53.300678 29120 mini_ranger_kms.cc:384] Added ranger policy
I20260809 07:54:53.300849 29120 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ranger-kms:/tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 07:54:53.300875 29120 mini_ranger_kms.cc:240] Using host: 127.28.112.20
I20260809 07:54:53.305363 29120 mini_ranger_kms.cc:276] Ranger KMS PID: 13594
I20260809 07:54:53.305508 29120 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.112.20:50141
Aug 09, 2026 7:54:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 7:54:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 7:54:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 7:54:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 7:54:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 7:54:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 07:54:55 dist-test-slave-vpvm krb5kdc[12980](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786262095, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.112.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:54:55.775714 29120 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.385s user 0.020s sys 0.194s
I20260809 07:54:55.775835 29120 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 07:54:55.775977 29120 mini_ranger_kms.cc:401] 127.28.112.20:50141/kms/v1/keys
Aug 09 07:54:56 dist-test-slave-vpvm krb5kdc[12980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262095, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.112.20@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
Aug 09 07:54:56 dist-test-slave-vpvm krb5kdc[12980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262095, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.112.20@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
Aug 09 07:54:58 dist-test-slave-vpvm krb5kdc[12980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:54:58.620609 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:40757
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:40757
--ranger_config_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.112.20:50141/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:54:58.734622 13687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:54:58.734848 13687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:54:58.734881 13687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:54:58.738896 13687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:54:58.738947 13687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:54:58.738965 13687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:54:58.738979 13687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:54:58.738993 13687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:54:58.744138 13687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.112.20:50141/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:40757
--ranger_config_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:40757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:54:58.745054 13687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:54:58.746008 13687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:54:58.754284 13697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:58.754331 13687 server_base.cc:1034] running on GCE node
W20260809 07:54:58.754294 13695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:58.754352 13701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:58.755071 13687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:54:58.756105 13687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:54:58.757262 13687 hybrid_clock.cc:648] HybridClock initialized: now 1786262098757243 us; error 30 us; skew 500 ppm
Aug 09 07:54:58 dist-test-slave-vpvm krb5kdc[12980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262098, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:54:58.764346 13687 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:54:58.765210 13687 webserver.cc:460] Webserver started at http://127.28.112.62:44905/ using document root <none> and password file <none>
I20260809 07:54:58.765718 13687 fs_manager.cc:359] Metadata directory not provided
I20260809 07:54:58.765776 13687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:54:58.766028 13687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 07:54:58 dist-test-slave-vpvm krb5kdc[12980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262098, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
I20260809 07:54:58.827097 13687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "9f2d8ce65926465fb7c35c8af5bd0457"
format_stamp: "Formatted at 2026-08-09 07:54:58 on dist-test-slave-vpvm"
server_key: "fb71e041bd71935d3a5bd64b8efb5ef6"
server_key_iv: "31e3277815cc58bc10451fa76a3a6a94"
server_key_version: "kuduclusterkey@0"
I20260809 07:54:58.827837 13687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "9f2d8ce65926465fb7c35c8af5bd0457"
format_stamp: "Formatted at 2026-08-09 07:54:58 on dist-test-slave-vpvm"
server_key: "fb71e041bd71935d3a5bd64b8efb5ef6"
server_key_iv: "31e3277815cc58bc10451fa76a3a6a94"
server_key_version: "kuduclusterkey@0"
I20260809 07:54:58.832890 13687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 07:54:58.850071 13714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:58.851433 13687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 07:54:58.851579 13687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "9f2d8ce65926465fb7c35c8af5bd0457"
format_stamp: "Formatted at 2026-08-09 07:54:58 on dist-test-slave-vpvm"
server_key: "fb71e041bd71935d3a5bd64b8efb5ef6"
server_key_iv: "31e3277815cc58bc10451fa76a3a6a94"
server_key_version: "kuduclusterkey@0"
I20260809 07:54:58.851713 13687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:54:58.877741 13687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:54:58.880559 13687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:54:58.880753 13687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:54:58.888682 13687 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:40757
I20260809 07:54:58.888731 13788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:40757 every 8 connection(s)
I20260809 07:54:58.889671 13687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:54:58.892508 13789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:54:58.898011 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 13687
I20260809 07:54:58.898146 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:54:58.898129 13789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457: Bootstrap starting.
I20260809 07:54:58.898445 29120 external_mini_cluster.cc:1428] Setting key d15bca6b975bb9771071fc61a4d174dc
I20260809 07:54:58.900740 13789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457: Neither blocks nor log segments found. Creating new log.
I20260809 07:54:58.901453 13789 log.cc:824] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457: Log is configured to *not* fsync() on all Append() calls
I20260809 07:54:58.903573 13789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457: No bootstrap required, opened a new log
I20260809 07:54:58.907462 13789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2d8ce65926465fb7c35c8af5bd0457" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40757 } }
I20260809 07:54:58.907742 13789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:54:58.907830 13789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f2d8ce65926465fb7c35c8af5bd0457, State: Initialized, Role: FOLLOWER
I20260809 07:54:58.908339 13789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2d8ce65926465fb7c35c8af5bd0457" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40757 } }
I20260809 07:54:58.908512 13789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:54:58.908627 13789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:54:58.908758 13789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457 [term 0 FOLLOWER]: Advancing to term 1
Aug 09 07:54:58 dist-test-slave-vpvm krb5kdc[12980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:54:58.909629 13789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2d8ce65926465fb7c35c8af5bd0457" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40757 } }
I20260809 07:54:58.910041 13789 leader_election.cc:302] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f2d8ce65926465fb7c35c8af5bd0457; no voters:
I20260809 07:54:58.910358 13789 leader_election.cc:288] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:54:58.910483 13795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:54:58.910706 13795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457 [term 1 LEADER]: Becoming Leader. State: Replica: 9f2d8ce65926465fb7c35c8af5bd0457, State: Running, Role: LEADER
I20260809 07:54:58.911046 13795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2d8ce65926465fb7c35c8af5bd0457" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40757 } }
I20260809 07:54:58.911669 13789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:54:58.912786 13795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f2d8ce65926465fb7c35c8af5bd0457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2d8ce65926465fb7c35c8af5bd0457" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40757 } } }
I20260809 07:54:58.913244 13795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457 [sys.catalog]: This master's current role is: LEADER
I20260809 07:54:58.913523 13795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f2d8ce65926465fb7c35c8af5bd0457. Latest consensus state: current_term: 1 leader_uuid: "9f2d8ce65926465fb7c35c8af5bd0457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2d8ce65926465fb7c35c8af5bd0457" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40757 } } }
I20260809 07:54:58.913664 13795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457 [sys.catalog]: This master's current role is: LEADER
I20260809 07:54:58.913988 13789 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 07:54:58.914732 13812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:54:58.915702 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:58.899926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41700 (local address 127.28.112.62:40757)
0809 07:54:58.900414 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:58.900435 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:58.900484 (+ 49us) server_negotiation.cc:407] Connection header received
0809 07:54:58.901351 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:58.901378 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:58.901806 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:58.902206 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:58.903115 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:58.904809 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:58.906057 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:58.906776 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:58.906958 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:58.909999 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:54:58.910007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:54:58.910026 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:54:58.910059 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:54:58.912177 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:54:58.912677 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:54:58.912684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:54:58.912690 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:54:58.912829 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:54:58.913238 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:54:58.913241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:54:58.913243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:54:58.913860 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:54:58.914141 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:58.914437 (+ 296us) server_negotiation.cc:299] Negotiation successful
0809 07:54:58.914724 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":140,"threads_started":1}
I20260809 07:54:58.922024 13812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:54:58.930743 13812 catalog_manager.cc:1347] Generated new cluster ID: 7a7945a007e54d8b8445ac43c91bdf28
I20260809 07:54:58.930815 13812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:54:58.946563 13812 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:54:58.948449 13812 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:54:58.960563 13812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457: Generated new TSK 0
I20260809 07:54:58.961565 13812 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
I20260809 07:54:59.700376 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.112.20:50141/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:40757
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:54:59.817157 13857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:54:59.817382 13857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:54:59.817438 13857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:54:59.821529 13857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:54:59.821596 13857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:54:59.821674 13857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:54:59.827286 13857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.112.20:50141/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:40757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:54:59.828293 13857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:54:59.829280 13857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:54:59.838845 13865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:59.838806 13869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:54:59.838809 13863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:54:59.839138 13857 server_base.cc:1034] running on GCE node
I20260809 07:54:59.839599 13857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:54:59.840246 13857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:54:59.841445 13857 hybrid_clock.cc:648] HybridClock initialized: now 1786262099841424 us; error 37 us; skew 500 ppm
Aug 09 07:54:59 dist-test-slave-vpvm krb5kdc[12980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262099, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:54:59.844393 13857 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:54:59.845436 13857 webserver.cc:460] Webserver started at http://127.28.112.1:43731/ using document root <none> and password file <none>
I20260809 07:54:59.846160 13857 fs_manager.cc:359] Metadata directory not provided
I20260809 07:54:59.846236 13857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:54:59.846438 13857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 07:54:59 dist-test-slave-vpvm krb5kdc[12980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262099, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
I20260809 07:54:59.862697 13857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "c3efc243a14440aaa4f20aa02577a09a"
format_stamp: "Formatted at 2026-08-09 07:54:59 on dist-test-slave-vpvm"
server_key: "baa31596378050fb26a40a0a70624a51"
server_key_iv: "b9f04fb315c0a2ec187364f29d26c10e"
server_key_version: "kuduclusterkey@0"
I20260809 07:54:59.863566 13857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "c3efc243a14440aaa4f20aa02577a09a"
format_stamp: "Formatted at 2026-08-09 07:54:59 on dist-test-slave-vpvm"
server_key: "baa31596378050fb26a40a0a70624a51"
server_key_iv: "b9f04fb315c0a2ec187364f29d26c10e"
server_key_version: "kuduclusterkey@0"
I20260809 07:54:59.869417 13857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 07:54:59.887127 13881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:59.888814 13857 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260809 07:54:59.889016 13857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "c3efc243a14440aaa4f20aa02577a09a"
format_stamp: "Formatted at 2026-08-09 07:54:59 on dist-test-slave-vpvm"
server_key: "baa31596378050fb26a40a0a70624a51"
server_key_iv: "b9f04fb315c0a2ec187364f29d26c10e"
server_key_version: "kuduclusterkey@0"
I20260809 07:54:59.889199 13857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:54:59.907070 13857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:54:59.910530 13857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:54:59.910766 13857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:54:59.911546 13857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:54:59.912632 13857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:54:59.912734 13857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:59.912853 13857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:54:59.912949 13857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:54:59.927327 13857 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:41749
I20260809 07:54:59.927596 14044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:41749 every 8 connection(s)
I20260809 07:54:59.928579 13857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:54:59.937501 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 13857
I20260809 07:54:59.937654 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:54:59.938060 29120 external_mini_cluster.cc:1428] Setting key 90893fbc1daa7ad10c8e20205a48607b
Aug 09 07:54:59 dist-test-slave-vpvm krb5kdc[12980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262099, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:54:59.948444 14052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:59.931614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:40757 (local address 127.28.112.1:55641)
0809 07:54:59.932972 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
0809 07:54:59.933040 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0809 07:54:59.934238 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:54:59.935131 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:54:59.935140 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:54:59.935520 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:54:59.936195 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:59.936216 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:59.937478 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:59.937482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:59.939078 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:54:59.939092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:54:59.939955 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:54:59.939960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:54:59.940167 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:59.940873 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:54:59.940897 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:54:59.942689 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:54:59.945109 (+ 2420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:54:59.945116 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:54:59.945132 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:54:59.945502 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:54:59.945851 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:54:59.945855 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:54:59.945857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:54:59.946079 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:54:59.946613 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:54:59.946620 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:54:59.946925 (+ 305us) client_negotiation.cc:769] Sending connection context
0809 07:54:59.947129 (+ 204us) client_negotiation.cc:240] Negotiation successful
0809 07:54:59.947449 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1117,"thread_start_us":162,"threads_started":1}
I20260809 07:54:59.949188 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:54:59.931051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:55641 (local address 127.28.112.62:40757)
0809 07:54:59.931384 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0809 07:54:59.931400 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 07:54:59.933888 (+ 2488us) server_negotiation.cc:407] Connection header received
0809 07:54:59.934624 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:54:59.934629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:54:59.934693 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:54:59.934802 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:54:59.936405 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:59.937290 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:59.939236 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:54:59.939637 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:54:59.939755 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:54:59.942904 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:54:59.942909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:54:59.942912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:54:59.942941 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:54:59.944947 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:54:59.945633 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:54:59.945637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:54:59.945639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:54:59.945737 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:54:59.946209 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:54:59.946212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:54:59.946214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:54:59.946471 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:54:59.946586 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 07:54:59.948884 (+ 2298us) server_negotiation.cc:299] Negotiation successful
0809 07:54:59.949025 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":126,"threads_started":1}
I20260809 07:54:59.956560 14046 heartbeater.cc:347] Connected to a master server at 127.28.112.62:40757
I20260809 07:54:59.956840 14046 heartbeater.cc:464] Registering TS with master...
I20260809 07:54:59.957440 14046 heartbeater.cc:511] Master 127.28.112.62:40757 requested a full tablet report, sending...
I20260809 07:54:59.959309 13738 ts_manager.cc:194] Registered new tserver with Master: c3efc243a14440aaa4f20aa02577a09a (127.28.112.1:41749)
I20260809 07:54:59.960875 13738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:55641
WARNING: no policy specified for kudu/127.28.112.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
I20260809 07:55:00.005271 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.112.20:50141/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:40757
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 07:55:00 dist-test-slave-vpvm krb5kdc[12980](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786262100, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 07:55:00.139889 14058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:00.140331 14058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:00.140450 14058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:00.145176 14058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:00.145255 14058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:00.145344 14058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:55:00.150905 14058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.112.20:50141/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:40757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:00.152011 14058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:00.153151 14058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:00.164664 14069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:00.164575 14073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:00.164839 14070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:00.165275 14058 server_base.cc:1034] running on GCE node
I20260809 07:55:00.165695 14058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:00.166363 14058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:00.173748 14058 hybrid_clock.cc:648] HybridClock initialized: now 1786262100173719 us; error 34 us; skew 500 ppm
Aug 09 07:55:00 dist-test-slave-vpvm krb5kdc[12980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262100, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:00.177565 14058 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:55:00.178932 14058 webserver.cc:460] Webserver started at http://127.28.112.2:46033/ using document root <none> and password file <none>
I20260809 07:55:00.179718 14058 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:00.179862 14058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:00.180182 14058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 07:55:00 dist-test-slave-vpvm krb5kdc[12980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262100, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
I20260809 07:55:00.201009 14058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "691e57d0d34a4bf5875d41a6c269c529"
format_stamp: "Formatted at 2026-08-09 07:55:00 on dist-test-slave-vpvm"
server_key: "26a6a7337b5b784bf6640ac4c20588d1"
server_key_iv: "b20583dfa454eaef8ae4220e74f9554e"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:00.201819 14058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "691e57d0d34a4bf5875d41a6c269c529"
format_stamp: "Formatted at 2026-08-09 07:55:00 on dist-test-slave-vpvm"
server_key: "26a6a7337b5b784bf6640ac4c20588d1"
server_key_iv: "b20583dfa454eaef8ae4220e74f9554e"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:00.211560 14058 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.002s
I20260809 07:55:00.229321 14090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:00.230991 14058 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 07:55:00.231128 14058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "691e57d0d34a4bf5875d41a6c269c529"
format_stamp: "Formatted at 2026-08-09 07:55:00 on dist-test-slave-vpvm"
server_key: "26a6a7337b5b784bf6640ac4c20588d1"
server_key_iv: "b20583dfa454eaef8ae4220e74f9554e"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:00.231269 14058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:00.245865 14058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:00.249213 14058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:00.251994 14058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:00.252727 14058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:55:00.253815 14058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:55:00.253875 14058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:00.253933 14058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:55:00.253978 14058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260809 07:55:00.264802 14058 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:45499
I20260809 07:55:00.264840 14266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:45499 every 8 connection(s)
I20260809 07:55:00.265743 14058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:55:00.275923 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 14058
I20260809 07:55:00.276075 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:55:00.276399 29120 external_mini_cluster.cc:1428] Setting key 0c8c8d1951715261dc4e20eee82fa2fb
Aug 09 07:55:00 dist-test-slave-vpvm krb5kdc[12980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262100, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:55:00.282925 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:00.270050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:50411 (local address 127.28.112.62:40757)
0809 07:55:00.270177 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:00.270182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:00.270680 (+ 498us) server_negotiation.cc:407] Connection header received
0809 07:55:00.271893 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:00.271897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:00.271957 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:00.272012 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:00.273663 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.274335 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.275489 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.275812 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.275931 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:00.278814 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:00.278818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:00.278821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:00.278847 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:00.280505 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:00.281106 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:00.281109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:00.281111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:00.281197 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:00.281592 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:00.281595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:00.281597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:00.281820 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:00.281892 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:00.282605 (+ 713us) server_negotiation.cc:299] Negotiation successful
0809 07:55:00.282734 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 07:55:00.284577 14270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:00.268906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:40757 (local address 127.28.112.2:50411)
0809 07:55:00.270512 (+ 1606us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:00.270570 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:00.271639 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:00.272424 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:00.272434 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:00.272776 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:00.273480 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:00.273502 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.274481 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.274484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:00.275365 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:00.275375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.276125 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.276130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:00.276327 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:00.276983 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:00.277002 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:00.278640 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:00.280633 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:00.280639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:00.280655 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:00.280994 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:00.281300 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:00.281303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:00.281305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:00.281481 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:00.282016 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:00.282023 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:00.282327 (+ 304us) client_negotiation.cc:769] Sending connection context
0809 07:55:00.283095 (+ 768us) client_negotiation.cc:240] Negotiation successful
0809 07:55:00.283454 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1368,"thread_start_us":134,"threads_started":1}
I20260809 07:55:00.285923 14267 heartbeater.cc:347] Connected to a master server at 127.28.112.62:40757
I20260809 07:55:00.286190 14267 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:00.286710 14267 heartbeater.cc:511] Master 127.28.112.62:40757 requested a full tablet report, sending...
I20260809 07:55:00.287778 13738 ts_manager.cc:194] Registered new tserver with Master: 691e57d0d34a4bf5875d41a6c269c529 (127.28.112.2:45499)
I20260809 07:55:00.288322 13738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:50411
WARNING: no policy specified for kudu/127.28.112.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
I20260809 07:55:00.308703 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.112.20:50141/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:40757
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:55:00.444240 14274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:00.444548 14274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:00.444608 14274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:00.449932 14274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:00.450031 14274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:00.450119 14274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:55:00.457581 14274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.112.20:50141/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:40757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:00.458963 14274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:00.460340 14274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:00.472807 14284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:00.473596 14281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:00.473973 14282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:00.474495 14274 server_base.cc:1034] running on GCE node
I20260809 07:55:00.475080 14274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:00.475842 14274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:00.477033 14274 hybrid_clock.cc:648] HybridClock initialized: now 1786262100477006 us; error 42 us; skew 500 ppm
Aug 09 07:55:00 dist-test-slave-vpvm krb5kdc[12980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262100, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:00.480685 14274 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:55:00.481791 14274 webserver.cc:460] Webserver started at http://127.28.112.3:34525/ using document root <none> and password file <none>
I20260809 07:55:00.482508 14274 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:00.482597 14274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:00.482846 14274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 07:55:00 dist-test-slave-vpvm krb5kdc[12980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262100, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
I20260809 07:55:00.506453 14274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "1e36de03c74b45de9655c36ff7910584"
format_stamp: "Formatted at 2026-08-09 07:55:00 on dist-test-slave-vpvm"
server_key: "8b2168fe04c733d97ad65175178d3ca0"
server_key_iv: "ae9eade6653e3c89615e1207ca23179d"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:00.507303 14274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "1e36de03c74b45de9655c36ff7910584"
format_stamp: "Formatted at 2026-08-09 07:55:00 on dist-test-slave-vpvm"
server_key: "8b2168fe04c733d97ad65175178d3ca0"
server_key_iv: "ae9eade6653e3c89615e1207ca23179d"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:00.512727 14274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 07:55:00.537737 14294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:00.539327 14274 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.001s
I20260809 07:55:00.539461 14274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "1e36de03c74b45de9655c36ff7910584"
format_stamp: "Formatted at 2026-08-09 07:55:00 on dist-test-slave-vpvm"
server_key: "8b2168fe04c733d97ad65175178d3ca0"
server_key_iv: "ae9eade6653e3c89615e1207ca23179d"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:00.539614 14274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:00.551151 14274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:00.554052 14274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:00.554222 14274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:00.554840 14274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:55:00.555763 14274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:55:00.555814 14274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:00.555867 14274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:55:00.555892 14274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:00.570449 14274 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:38961
I20260809 07:55:00.570683 14468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:38961 every 8 connection(s)
I20260809 07:55:00.571851 14274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:55:00.572392 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 14274
I20260809 07:55:00.572521 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:55:00.572845 29120 external_mini_cluster.cc:1428] Setting key a10b42d42eed19f350fc7b5f3da7168a
Aug 09 07:55:00 dist-test-slave-vpvm krb5kdc[12980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262100, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:55:00.596323 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:00.576624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:37969 (local address 127.28.112.62:40757)
0809 07:55:00.576749 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:00.576754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:00.582502 (+ 5748us) server_negotiation.cc:407] Connection header received
0809 07:55:00.583918 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:00.583922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:00.583978 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:00.584026 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:00.585777 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.586587 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.588250 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.588580 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.588658 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:00.591766 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:00.591770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:00.591773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:00.591801 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:00.593754 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:00.594400 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:00.594404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:00.594406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:00.594494 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:00.594891 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:00.594894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:00.594896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:00.595131 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:00.595214 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:00.596038 (+ 824us) server_negotiation.cc:299] Negotiation successful
0809 07:55:00.596156 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 07:55:00.598627 14472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:00.577123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:40757 (local address 127.28.112.3:37969)
0809 07:55:00.582339 (+ 5216us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:00.582397 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:00.583674 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:00.584440 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:00.584449 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:00.584873 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:00.585613 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:00.585634 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.586739 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.586743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:00.588117 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:00.588129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.588855 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.588859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:00.589070 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:00.589768 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:00.589789 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:00.591582 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:00.593890 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:00.593897 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:00.593913 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:00.594282 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:00.594606 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:00.594609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:00.594611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:00.594783 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:00.595288 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:00.595294 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:00.595750 (+ 456us) client_negotiation.cc:769] Sending connection context
0809 07:55:00.596816 (+ 1066us) client_negotiation.cc:240] Negotiation successful
0809 07:55:00.597299 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4946,"thread_start_us":246,"threads_started":1}
I20260809 07:55:00.600373 14470 heartbeater.cc:347] Connected to a master server at 127.28.112.62:40757
I20260809 07:55:00.600610 14470 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:00.601162 14470 heartbeater.cc:511] Master 127.28.112.62:40757 requested a full tablet report, sending...
I20260809 07:55:00.602278 13738 ts_manager.cc:194] Registered new tserver with Master: 1e36de03c74b45de9655c36ff7910584 (127.28.112.3:38961)
I20260809 07:55:00.603050 13738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:37969
Aug 09 07:55:00 dist-test-slave-vpvm krb5kdc[12980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262100, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
I20260809 07:55:00.611579 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 07:55:00.625917 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:00.614030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41714 (local address 127.28.112.62:40757)
0809 07:55:00.614151 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:00.614156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:00.614923 (+ 767us) server_negotiation.cc:407] Connection header received
0809 07:55:00.615151 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:00.615154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:00.615209 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:00.615270 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:00.616397 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.617220 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.618810 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.619158 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.619234 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:00.621020 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:00.621024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:00.621027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:00.621054 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:00.622676 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:00.624387 (+ 1711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:00.624390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:00.624392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:00.624484 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:00.624832 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:00.624836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:00.624837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:00.625091 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:00.625199 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:00.625604 (+ 405us) server_negotiation.cc:299] Negotiation successful
0809 07:55:00.625730 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 07:55:00.630074 13738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:55:00.633212 13738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:55:00.656996 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:00.646215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:38961 (local address 127.0.0.1:53498)
0809 07:55:00.647298 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:00.647356 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:00.647597 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:00.648742 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:00.648891 (+ 149us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:00.648923 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:55:00.649060 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:00.649067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.652463 (+ 3396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.652467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:00.653825 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:00.653837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.655973 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.655982 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:00.656136 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:00.656186 (+ 50us) client_negotiation.cc:769] Sending connection context
0809 07:55:00.656277 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 07:55:00.656402 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":1,"thread_start_us":108,"threads_started":1}
I20260809 07:55:00.657860 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:00.646672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.28.112.3:38961)
0809 07:55:00.648239 (+ 1567us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:00.648252 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:00.648277 (+ 25us) server_negotiation.cc:407] Connection header received
0809 07:55:00.648355 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:00.648361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:00.648584 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:00.648761 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:55:00.649330 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.652345 (+ 3015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.654025 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.654845 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.657223 (+ 2378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:00.657392 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:00.657451 (+ 59us) server_negotiation.cc:299] Negotiation successful
0809 07:55:00.657549 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1461,"thread_start_us":180,"threads_started":1}
I20260809 07:55:00.660830 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:00.644162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:45499 (local address 127.0.0.1:54726)
0809 07:55:00.648780 (+ 4618us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:00.648794 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:00.649170 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:00.651203 (+ 2033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:00.651206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:00.651223 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:55:00.651308 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:00.651313 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.655748 (+ 4435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.655752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:00.658228 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:00.658240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.659681 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.659685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:00.659774 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:00.659799 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 07:55:00.660624 (+ 825us) client_negotiation.cc:240] Negotiation successful
0809 07:55:00.660702 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2582,"mutex_wait_us":478,"thread_start_us":160,"threads_started":1}
I20260809 07:55:00.661283 14489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:00.644293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54726 (local address 127.28.112.2:45499)
0809 07:55:00.648607 (+ 4314us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:00.648619 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:00.649134 (+ 515us) server_negotiation.cc:407] Connection header received
0809 07:55:00.649719 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:00.649725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:00.649910 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:00.651051 (+ 1141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:55:00.651811 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.652989 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.658396 (+ 5407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.659373 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.659476 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:00.659616 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:00.661022 (+ 1406us) server_negotiation.cc:299] Negotiation successful
0809 07:55:00.661146 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4202,"thread_start_us":130,"threads_started":1}
I20260809 07:55:00.661324 14342 tablet_service.cc:1467] Processing CreateTablet for tablet db3dfaba9ff14c40bbb4c33acb3160c7 (DEFAULT_TABLE table=test-table [id=7b1e3650bb9a426dbb6070820ad2f397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:55:00.662528 14342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3dfaba9ff14c40bbb4c33acb3160c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:55:00.669076 14169 tablet_service.cc:1467] Processing CreateTablet for tablet db3dfaba9ff14c40bbb4c33acb3160c7 (DEFAULT_TABLE table=test-table [id=7b1e3650bb9a426dbb6070820ad2f397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:55:00.669590 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:00.647533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:41749 (local address 127.0.0.1:35476)
0809 07:55:00.647877 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:00.647891 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:00.647993 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:00.665151 (+ 17158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:00.665155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:00.665174 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:55:00.665258 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:00.665263 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.666694 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.666698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:00.667787 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:00.667798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.669161 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.669165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:00.669253 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:00.669287 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 07:55:00.669366 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 07:55:00.669465 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":93,"threads_started":1}
I20260809 07:55:00.670238 14494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:00.647709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.28.112.1:41749)
0809 07:55:00.658560 (+ 10851us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:00.658567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:00.664506 (+ 5939us) server_negotiation.cc:407] Connection header received
0809 07:55:00.664631 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:00.664637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:00.664842 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:00.664992 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:55:00.665399 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.666562 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.667946 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.668763 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.668865 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:00.669077 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:00.670001 (+ 924us) server_negotiation.cc:299] Negotiation successful
0809 07:55:00.670102 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10716,"thread_start_us":102,"threads_started":1}
I20260809 07:55:00.670298 14169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3dfaba9ff14c40bbb4c33acb3160c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:55:00.677295 13954 tablet_service.cc:1467] Processing CreateTablet for tablet db3dfaba9ff14c40bbb4c33acb3160c7 (DEFAULT_TABLE table=test-table [id=7b1e3650bb9a426dbb6070820ad2f397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:55:00.678532 13954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3dfaba9ff14c40bbb4c33acb3160c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:55:00.691344 14496 tablet_bootstrap.cc:492] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584: Bootstrap starting.
I20260809 07:55:00.695475 14501 tablet_bootstrap.cc:492] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529: Bootstrap starting.
I20260809 07:55:00.696300 14496 tablet_bootstrap.cc:654] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584: Neither blocks nor log segments found. Creating new log.
I20260809 07:55:00.697454 14496 log.cc:824] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:00.698228 14501 tablet_bootstrap.cc:654] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529: Neither blocks nor log segments found. Creating new log.
I20260809 07:55:00.698504 14495 tablet_bootstrap.cc:492] T db3dfaba9ff14c40bbb4c33acb3160c7 P c3efc243a14440aaa4f20aa02577a09a: Bootstrap starting.
I20260809 07:55:00.699083 14501 log.cc:824] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:00.700863 14495 tablet_bootstrap.cc:654] T db3dfaba9ff14c40bbb4c33acb3160c7 P c3efc243a14440aaa4f20aa02577a09a: Neither blocks nor log segments found. Creating new log.
I20260809 07:55:00.701747 14495 log.cc:824] T db3dfaba9ff14c40bbb4c33acb3160c7 P c3efc243a14440aaa4f20aa02577a09a: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:00.704118 14495 tablet_bootstrap.cc:492] T db3dfaba9ff14c40bbb4c33acb3160c7 P c3efc243a14440aaa4f20aa02577a09a: No bootstrap required, opened a new log
I20260809 07:55:00.704346 14495 ts_tablet_manager.cc:1397] T db3dfaba9ff14c40bbb4c33acb3160c7 P c3efc243a14440aaa4f20aa02577a09a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 07:55:00.707700 14501 tablet_bootstrap.cc:492] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529: No bootstrap required, opened a new log
I20260809 07:55:00.707894 14501 ts_tablet_manager.cc:1397] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260809 07:55:00.710592 14495 raft_consensus.cc:348] T db3dfaba9ff14c40bbb4c33acb3160c7 P c3efc243a14440aaa4f20aa02577a09a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691e57d0d34a4bf5875d41a6c269c529" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45499 } } peers { permanent_uuid: "1e36de03c74b45de9655c36ff7910584" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38961 } } peers { permanent_uuid: "c3efc243a14440aaa4f20aa02577a09a" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41749 } }
I20260809 07:55:00.711000 14495 raft_consensus.cc:374] T db3dfaba9ff14c40bbb4c33acb3160c7 P c3efc243a14440aaa4f20aa02577a09a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:55:00.711126 14495 raft_consensus.cc:729] T db3dfaba9ff14c40bbb4c33acb3160c7 P c3efc243a14440aaa4f20aa02577a09a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3efc243a14440aaa4f20aa02577a09a, State: Initialized, Role: FOLLOWER
I20260809 07:55:00.711756 14495 consensus_queue.cc:260] T db3dfaba9ff14c40bbb4c33acb3160c7 P c3efc243a14440aaa4f20aa02577a09a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691e57d0d34a4bf5875d41a6c269c529" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45499 } } peers { permanent_uuid: "1e36de03c74b45de9655c36ff7910584" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38961 } } peers { permanent_uuid: "c3efc243a14440aaa4f20aa02577a09a" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41749 } }
I20260809 07:55:00.712997 14496 tablet_bootstrap.cc:492] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584: No bootstrap required, opened a new log
I20260809 07:55:00.713236 14496 ts_tablet_manager.cc:1397] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.000s
I20260809 07:55:00.713280 14495 ts_tablet_manager.cc:1428] T db3dfaba9ff14c40bbb4c33acb3160c7 P c3efc243a14440aaa4f20aa02577a09a: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260809 07:55:00.713780 14046 heartbeater.cc:503] Master 127.28.112.62:40757 was elected leader, sending a full tablet report...
I20260809 07:55:00.719185 14496 raft_consensus.cc:348] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691e57d0d34a4bf5875d41a6c269c529" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45499 } } peers { permanent_uuid: "1e36de03c74b45de9655c36ff7910584" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38961 } } peers { permanent_uuid: "c3efc243a14440aaa4f20aa02577a09a" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41749 } }
I20260809 07:55:00.719409 14496 raft_consensus.cc:374] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:55:00.719491 14496 raft_consensus.cc:729] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e36de03c74b45de9655c36ff7910584, State: Initialized, Role: FOLLOWER
I20260809 07:55:00.719913 14496 consensus_queue.cc:260] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691e57d0d34a4bf5875d41a6c269c529" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45499 } } peers { permanent_uuid: "1e36de03c74b45de9655c36ff7910584" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38961 } } peers { permanent_uuid: "c3efc243a14440aaa4f20aa02577a09a" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41749 } }
I20260809 07:55:00.720587 14470 heartbeater.cc:503] Master 127.28.112.62:40757 was elected leader, sending a full tablet report...
I20260809 07:55:00.720943 14496 ts_tablet_manager.cc:1428] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260809 07:55:00.726662 14501 raft_consensus.cc:348] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691e57d0d34a4bf5875d41a6c269c529" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45499 } } peers { permanent_uuid: "1e36de03c74b45de9655c36ff7910584" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38961 } } peers { permanent_uuid: "c3efc243a14440aaa4f20aa02577a09a" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41749 } }
I20260809 07:55:00.726897 14501 raft_consensus.cc:374] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:55:00.726974 14501 raft_consensus.cc:729] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691e57d0d34a4bf5875d41a6c269c529, State: Initialized, Role: FOLLOWER
I20260809 07:55:00.727409 14501 consensus_queue.cc:260] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691e57d0d34a4bf5875d41a6c269c529" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45499 } } peers { permanent_uuid: "1e36de03c74b45de9655c36ff7910584" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38961 } } peers { permanent_uuid: "c3efc243a14440aaa4f20aa02577a09a" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41749 } }
I20260809 07:55:00.728149 14267 heartbeater.cc:503] Master 127.28.112.62:40757 was elected leader, sending a full tablet report...
I20260809 07:55:00.728428 14501 ts_tablet_manager.cc:1428] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
W20260809 07:55:00.767453 14268 tablet.cc:2369] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:55:00.815465 14508 raft_consensus.cc:482] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:55:00.815707 14508 raft_consensus.cc:504] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691e57d0d34a4bf5875d41a6c269c529" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45499 } } peers { permanent_uuid: "1e36de03c74b45de9655c36ff7910584" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38961 } } peers { permanent_uuid: "c3efc243a14440aaa4f20aa02577a09a" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41749 } }
I20260809 07:55:00.817059 14508 leader_election.cc:288] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 691e57d0d34a4bf5875d41a6c269c529 (127.28.112.2:45499), c3efc243a14440aaa4f20aa02577a09a (127.28.112.1:41749)
I20260809 07:55:00.822628 14472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:00.817588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:45499 (local address 127.28.112.3:39269)
0809 07:55:00.818160 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:00.818175 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:00.818268 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:00.818659 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:00.818663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:00.818683 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:55:00.818804 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:00.818811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.819868 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.819872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:00.821128 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:00.821140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.822186 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.822190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:00.822293 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:00.822320 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 07:55:00.822400 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 07:55:00.822480 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489}
I20260809 07:55:00.822824 14494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:00.818534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:39265 (local address 127.28.112.1:41749)
0809 07:55:00.818658 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:00.818663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:00.818872 (+ 209us) server_negotiation.cc:407] Connection header received
0809 07:55:00.819038 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:00.819042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:00.819087 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:00.819130 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:55:00.819501 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.820199 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.821591 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.822172 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.822258 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:00.822305 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:00.822636 (+ 331us) server_negotiation.cc:299] Negotiation successful
0809 07:55:00.822711 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 07:55:00.823136 14489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:00.818156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:39269 (local address 127.28.112.2:45499)
0809 07:55:00.818283 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:00.818289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:00.818309 (+ 20us) server_negotiation.cc:407] Connection header received
0809 07:55:00.818372 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:00.818375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:00.818428 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:00.818521 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:55:00.818965 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.819727 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.821301 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.821899 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.822059 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:00.822117 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:00.822915 (+ 798us) server_negotiation.cc:299] Negotiation successful
0809 07:55:00.823013 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 07:55:00.823751 14196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3dfaba9ff14c40bbb4c33acb3160c7" candidate_uuid: "1e36de03c74b45de9655c36ff7910584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691e57d0d34a4bf5875d41a6c269c529" is_pre_election: true
I20260809 07:55:00.824087 14196 raft_consensus.cc:2454] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e36de03c74b45de9655c36ff7910584 in term 0.
I20260809 07:55:00.824618 14299 leader_election.cc:302] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e36de03c74b45de9655c36ff7910584, 691e57d0d34a4bf5875d41a6c269c529; no voters:
I20260809 07:55:00.824812 14510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:00.817796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:41749 (local address 127.28.112.3:39265)
0809 07:55:00.818800 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:00.818812 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:00.818934 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:00.819249 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:00.819251 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:00.819267 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:55:00.819378 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:00.819383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.820343 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.820346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:00.821444 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:00.821463 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.822405 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.822409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:00.822510 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:00.822532 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 07:55:00.824448 (+ 1916us) client_negotiation.cc:240] Negotiation successful
0809 07:55:00.824536 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"thread_start_us":101,"threads_started":1}
I20260809 07:55:00.824952 14508 raft_consensus.cc:2781] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:55:00.825033 14508 raft_consensus.cc:482] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:55:00.825086 14508 raft_consensus.cc:3037] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:55:00.825541 13971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3dfaba9ff14c40bbb4c33acb3160c7" candidate_uuid: "1e36de03c74b45de9655c36ff7910584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3efc243a14440aaa4f20aa02577a09a" is_pre_election: true
I20260809 07:55:00.825809 13971 raft_consensus.cc:2454] T db3dfaba9ff14c40bbb4c33acb3160c7 P c3efc243a14440aaa4f20aa02577a09a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e36de03c74b45de9655c36ff7910584 in term 0.
W20260809 07:55:00.826149 14471 tablet.cc:2369] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:55:00.826437 14508 raft_consensus.cc:504] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691e57d0d34a4bf5875d41a6c269c529" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45499 } } peers { permanent_uuid: "1e36de03c74b45de9655c36ff7910584" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38961 } } peers { permanent_uuid: "c3efc243a14440aaa4f20aa02577a09a" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41749 } }
I20260809 07:55:00.826779 14508 leader_election.cc:288] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [CANDIDATE]: Term 1 election: Requested vote from peers 691e57d0d34a4bf5875d41a6c269c529 (127.28.112.2:45499), c3efc243a14440aaa4f20aa02577a09a (127.28.112.1:41749)
I20260809 07:55:00.827368 13985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3dfaba9ff14c40bbb4c33acb3160c7" candidate_uuid: "1e36de03c74b45de9655c36ff7910584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3efc243a14440aaa4f20aa02577a09a"
I20260809 07:55:00.827502 13985 raft_consensus.cc:3037] T db3dfaba9ff14c40bbb4c33acb3160c7 P c3efc243a14440aaa4f20aa02577a09a [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:55:00.828796 13985 raft_consensus.cc:2454] T db3dfaba9ff14c40bbb4c33acb3160c7 P c3efc243a14440aaa4f20aa02577a09a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e36de03c74b45de9655c36ff7910584 in term 1.
I20260809 07:55:00.829099 14196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3dfaba9ff14c40bbb4c33acb3160c7" candidate_uuid: "1e36de03c74b45de9655c36ff7910584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691e57d0d34a4bf5875d41a6c269c529"
I20260809 07:55:00.829226 14196 raft_consensus.cc:3037] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:55:00.830549 14196 raft_consensus.cc:2454] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e36de03c74b45de9655c36ff7910584 in term 1.
I20260809 07:55:00.831498 14299 leader_election.cc:302] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e36de03c74b45de9655c36ff7910584, 691e57d0d34a4bf5875d41a6c269c529; no voters:
I20260809 07:55:00.831688 14508 raft_consensus.cc:2781] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:55:00.831918 14508 raft_consensus.cc:686] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [term 1 LEADER]: Becoming Leader. State: Replica: 1e36de03c74b45de9655c36ff7910584, State: Running, Role: LEADER
I20260809 07:55:00.832355 14508 consensus_queue.cc:237] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691e57d0d34a4bf5875d41a6c269c529" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45499 } } peers { permanent_uuid: "1e36de03c74b45de9655c36ff7910584" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38961 } } peers { permanent_uuid: "c3efc243a14440aaa4f20aa02577a09a" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41749 } }
I20260809 07:55:00.837875 13734 catalog_manager.cc:5495] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e36de03c74b45de9655c36ff7910584 (127.28.112.3). New cstate: current_term: 1 leader_uuid: "1e36de03c74b45de9655c36ff7910584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691e57d0d34a4bf5875d41a6c269c529" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e36de03c74b45de9655c36ff7910584" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3efc243a14440aaa4f20aa02577a09a" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41749 } health_report { overall_health: UNKNOWN } } }
I20260809 07:55:00.904502 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:00.887707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53506 (local address 127.28.112.3:38961)
0809 07:55:00.887847 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:00.887852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:00.887868 (+ 16us) server_negotiation.cc:407] Connection header received
0809 07:55:00.887960 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:00.887963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:00.888035 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:00.888130 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:55:00.888510 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.889276 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.897265 (+ 7989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:00.897609 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:00.897720 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:00.898196 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:55:00.898799 (+ 603us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:55:00.898920 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:00.904267 (+ 5347us) server_negotiation.cc:299] Negotiation successful
0809 07:55:00.904344 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 07:55:00.920833 13985 raft_consensus.cc:1261] T db3dfaba9ff14c40bbb4c33acb3160c7 P c3efc243a14440aaa4f20aa02577a09a [term 1 FOLLOWER]: Refusing update from remote peer 1e36de03c74b45de9655c36ff7910584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:55:00.921705 14511 consensus_queue.cc:1035] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3efc243a14440aaa4f20aa02577a09a" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 07:55:00.930527 14048 tablet.cc:2369] T db3dfaba9ff14c40bbb4c33acb3160c7 P c3efc243a14440aaa4f20aa02577a09a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:55:00.933569 14196 raft_consensus.cc:1261] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529 [term 1 FOLLOWER]: Refusing update from remote peer 1e36de03c74b45de9655c36ff7910584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:55:00.936936 14508 consensus_queue.cc:1035] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "691e57d0d34a4bf5875d41a6c269c529" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 45499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:55:00.938189 14514 mvcc.cc:205] Tried to move back new op lower bound from 7316529565362515968 to 7316529565012791296. Current Snapshot: MvccSnapshot[applied={T|T < 7316529565362515968}]
Aug 09 07:55:00 dist-test-slave-vpvm krb5kdc[12980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262100, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
I20260809 07:55:00.954352 13734 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:55:00.954523 13734 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:55:00.957345 13734 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f2d8ce65926465fb7c35c8af5bd0457: Sending DeleteTablet for 3 replicas of tablet db3dfaba9ff14c40bbb4c33acb3160c7
I20260809 07:55:00.958488 14342 tablet_service.cc:1514] Processing DeleteTablet for tablet db3dfaba9ff14c40bbb4c33acb3160c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:55:00 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:53498
I20260809 07:55:00.959476 14169 tablet_service.cc:1514] Processing DeleteTablet for tablet db3dfaba9ff14c40bbb4c33acb3160c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:55:00 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:54726
I20260809 07:55:00.959975 13954 tablet_service.cc:1514] Processing DeleteTablet for tablet db3dfaba9ff14c40bbb4c33acb3160c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:55:00 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:35476
I20260809 07:55:00.961022 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 13857
I20260809 07:55:00.973007 14523 mvcc.cc:205] Tried to move back new op lower bound from 7316529565362515968 to 7316529565012791296. Current Snapshot: MvccSnapshot[applied={T|T < 7316529565362515968}]
I20260809 07:55:00.973376 14528 tablet_replica.cc:330] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584: stopping tablet replica
I20260809 07:55:00.975275 14530 tablet_replica.cc:330] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529: stopping tablet replica
I20260809 07:55:00.978348 14530 raft_consensus.cc:2229] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:55:00.978749 14528 raft_consensus.cc:2229] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:55:00.979079 14530 raft_consensus.cc:2258] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:55:00.979527 14528 raft_consensus.cc:2258] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:55:00.981467 14528 ts_tablet_manager.cc:1905] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:55:00.988152 14530 ts_tablet_manager.cc:1905] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:55:00.992954 14528 ts_tablet_manager.cc:1918] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:55:00.999259 14530 ts_tablet_manager.cc:1918] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:55:00.999475 14528 log.cc:1196] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/db3dfaba9ff14c40bbb4c33acb3160c7
I20260809 07:55:00.999909 14528 ts_tablet_manager.cc:1939] T db3dfaba9ff14c40bbb4c33acb3160c7 P 1e36de03c74b45de9655c36ff7910584: Deleting consensus metadata
I20260809 07:55:01.000090 14530 log.cc:1196] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/db3dfaba9ff14c40bbb4c33acb3160c7
I20260809 07:55:01.000481 14530 ts_tablet_manager.cc:1939] T db3dfaba9ff14c40bbb4c33acb3160c7 P 691e57d0d34a4bf5875d41a6c269c529: Deleting consensus metadata
I20260809 07:55:01.002005 13715 catalog_manager.cc:4841] TS 1e36de03c74b45de9655c36ff7910584 (127.28.112.3:38961): tablet db3dfaba9ff14c40bbb4c33acb3160c7 (table test-table [id=7b1e3650bb9a426dbb6070820ad2f397]) successfully deleted
I20260809 07:55:01.010429 13718 catalog_manager.cc:4841] TS 691e57d0d34a4bf5875d41a6c269c529 (127.28.112.2:45499): tablet db3dfaba9ff14c40bbb4c33acb3160c7 (table test-table [id=7b1e3650bb9a426dbb6070820ad2f397]) successfully deleted
W20260809 07:55:01.011128 13718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.112.1:41749 (error 108)
I20260809 07:55:01.012835 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:01.012460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:41749 (local address 127.0.0.1:35488)
0809 07:55:01.012657 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:01.012747 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:41749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260809 07:55:01.013173 13718 catalog_manager.cc:4584] TS c3efc243a14440aaa4f20aa02577a09a (127.28.112.1:41749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db3dfaba9ff14c40bbb4c33acb3160c7: Network error: Client connection negotiation failed: client connection to 127.28.112.1:41749: connect: Connection refused (error 111)
I20260809 07:55:01.013578 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 14058
I20260809 07:55:01.063426 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 14274
I20260809 07:55:01.068063 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:01.067690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:41749 (local address 127.0.0.1:35498)
0809 07:55:01.067910 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:01.067968 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:41749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260809 07:55:01.068990 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:01.068746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:41749 (local address 127.0.0.1:35504)
0809 07:55:01.068878 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:01.068922 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:41749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260809 07:55:01.077323 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 13687
I20260809 07:55:01.090621 29120 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 07:55:01.434561 29120 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 07:55:01.434691 29120 mini_ranger.cc:66] Stopping Ranger...
I20260809 07:55:01.778964 29120 mini_ranger.cc:68] Stopped Ranger
2026-08-09 07:55:01.779 UTC [13024] LOG: received smart shutdown request
2026-08-09 07:55:01.782 UTC [13024] LOG: background worker "logical replication launcher" (PID 13039) exited with exit code 1
2026-08-09 07:55:01.783 UTC [13030] LOG: shutting down
2026-08-09 07:55:01.929 UTC [13024] LOG: database system is shut down
2026-08-09T07:55:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45838 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:55:02 dist-test-slave-vpvm krb5kdc[14549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:55:02 dist-test-slave-vpvm krb5kdc[14549](info): set up 2 sockets
Aug 09 07:55:02 dist-test-slave-vpvm krb5kdc[14549](info): commencing operation
krb5kdc: starting...
W20260809 07:55:03.156867 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:55:03 dist-test-slave-vpvm krb5kdc[14549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:55:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:55:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.112.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.112.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.112.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/rangeradmin_127.28.112.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.112.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/rangeradmin_127.28.112.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.112.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.112.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.112.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/rangerlookup_127.28.112.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.112.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/rangerlookup_127.28.112.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.112.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.112.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.112.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/HTTP_127.28.112.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.112.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/HTTP_127.28.112.20@KRBTEST.COM.keytab.
I20260809 07:55:03.312322 29120 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task4dUq1L/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/postgres -l logfile start
2026-08-09 07:55:05.606 UTC [14602] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 07:55:05.606 UTC [14602] LOG: listening on IPv4 address "127.28.112.20", port 50573
2026-08-09 07:55:05.609 UTC [14602] LOG: listening on Unix socket "/tmp/.s.PGSQL.50573"
2026-08-09 07:55:05.629 UTC [14609] LOG: database system was shut down at 2026-08-09 07:55:04 UTC
2026-08-09 07:55:05.632 UTC [14602] LOG: database system is ready to accept connections
I20260809 07:55:06.677160 29120 mini_postgres.cc:96] Postgres bound to 50573
2026-08-09 07:55:06.684 UTC [14622] FATAL: database "slave" does not exist
127.28.112.20:50573 - accepting connections
I20260809 07:55:06.684623 29120 mini_ranger.cc:161] Starting Ranger...
I20260809 07:55:06.699326 29120 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 07:55:07.128497 29120 mini_ranger.cc:87] Created ranger Postgres database
I20260809 07:55:07.128578 29120 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ranger-admin
2026-08-09 07:55:07,399 [I] DB FLAVOR :POSTGRES
2026-08-09 07:55:07,399 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 07:55:07,399 [I] Checking connection..
2026-08-09 07:55:07,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 07:55:07,629 [I] Checking connection passed.
2026-08-09 07:55:07,629 [I] --------- Verifying version history table ---------
2026-08-09 07:55:07,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 07:55:07,881 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 07:55:07,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 07:55:08,122 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 07:55:08,122 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 07:55:08,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 07:55:08.388 UTC [14718] WARNING: there is no transaction in progress
2026-08-09 07:55:08,397 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 07:55:08,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 07:55:08,648 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 07:55:08,649 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 07:55:08,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 07:55:08,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 07:55:09,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vpvm.c.gcp-upstream.internal','N') ;"
2026-08-09 07:55:09,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 07:55:09,604 [I] Table x_portal_user does not exist in database ranger
2026-08-09 07:55:09,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 07:55:09,860 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 07:55:09,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 07:55:10,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 07:55:10,360 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 07:55:10,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 07:55:11.257 UTC [14884] WARNING: there is no transaction in progress
2026-08-09 07:55:11.272 UTC [14884] WARNING: there is no transaction in progress
2026-08-09 07:55:11.287 UTC [14884] WARNING: there is no transaction in progress
2026-08-09 07:55:11.301 UTC [14884] WARNING: there is no transaction in progress
2026-08-09 07:55:11.315 UTC [14884] WARNING: there is no transaction in progress
2026-08-09 07:55:11.331 UTC [14884] WARNING: there is no transaction in progress
2026-08-09 07:55:11.381 UTC [14884] WARNING: there is no transaction in progress
2026-08-09 07:55:11.387 UTC [14884] WARNING: there is no transaction in progress
2026-08-09 07:55:11.399 UTC [14884] WARNING: there is no transaction in progress
2026-08-09 07:55:11.409 UTC [14884] WARNING: there is no transaction in progress
2026-08-09 07:55:11.418 UTC [14884] WARNING: there is no transaction in progress
2026-08-09 07:55:11.430 UTC [14884] WARNING: there is no transaction in progress
2026-08-09 07:55:11.439 UTC [14884] WARNING: there is no transaction in progress
2026-08-09 07:55:11.444 UTC [14884] WARNING: there is no transaction in progress
2026-08-09 07:55:11.450 UTC [14884] WARNING: there is no transaction in progress
2026-08-09 07:55:11.457 UTC [14884] WARNING: there is no transaction in progress
2026-08-09 07:55:12,028 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 07:55:12,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 07:55:12,275 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 07:55:12,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 07:55:12,509 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 07:55:12,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 07:55:12,763 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 07:55:12,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 07:55:13,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 07:55:13,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vpvm.c.gcp-upstream.internal';"
2026-08-09 07:55:13,502 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 07:55:13,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 07:55:13,758 [I] DB_PATCHES have already been applied
I20260809 07:55:13.764390 29120 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ranger-admin:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task4dUq1L/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 07:55:13.764454 29120 mini_ranger.cc:193] Using host: 127.28.112.20
I20260809 07:55:13.768298 29120 mini_ranger.cc:227] Ranger admin URL: http://127.28.112.20:52435
Aug 09, 2026 7:55:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 7:55:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 7:55:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 7:55:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 7:55:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 7:55:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 7:55:14 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 07:55:14.980083 29120 mini_ranger.cc:160] Time spent starting Ranger: real 8.295s user 0.000s sys 0.010s
Aug 09, 2026 7:55:34 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 7:55:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 09, 2026 7:55:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 7:55:34 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 7:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 7:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 7:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 7:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 7:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 7:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 7:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 7:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 7:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 7:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 7:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 7:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 7:55:34 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 7:55:35 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 07:55:36.260535 29120 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.112.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.112.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.112.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/rangerkms_127.28.112.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.112.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/rangerkms_127.28.112.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.112.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.112.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.112.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/HTTP_127.28.112.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.112.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/HTTP_127.28.112.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 07:55:36 dist-test-slave-vpvm krb5kdc[14549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262136, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 07:55:36.342576 29120 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 07:55:36.352550 29120 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 07:55:37.133541 29120 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 07:55:37.133616 29120 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ranger-kms
I20260809 07:55:37.133630 29120 mini_ranger_kms.cc:227] Using postgres at 127.28.112.20:50573
2026-08-09 07:55:37,295 [I] DB FLAVOR :POSTGRES
2026-08-09 07:55:37,296 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 07:55:37,296 [I] Checking connection
2026-08-09 07:55:37,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 07:55:37,549 [I] connection success
2026-08-09 07:55:37,549 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 07:55:37,549 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 07:55:37,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 07:55:37,774 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 07:55:37,774 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 07:55:37,774 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 07:55:37,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.112.20:50573/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 07:55:38,029 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 07:55:38.504503 29120 mini_ranger_kms.cc:310] Created kms service
I20260809 07:55:38.703881 29120 mini_ranger_kms.cc:326] Created kudu user
I20260809 07:55:38.744926 29120 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 07:55:38 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262136, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
I20260809 07:55:39.112051 29120 mini_ranger_kms.cc:384] Added ranger policy
I20260809 07:55:39.112314 29120 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ranger-kms:/tmp/dist-test-task4dUq1L/build/debug/bin/postgresql.jar:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task4dUq1L/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 07:55:39.112347 29120 mini_ranger_kms.cc:240] Using host: 127.28.112.20
I20260809 07:55:39.117554 29120 mini_ranger_kms.cc:276] Ranger KMS PID: 15162
I20260809 07:55:39.117624 29120 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.112.20:60007
Aug 09, 2026 7:55:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 7:55:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 7:55:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 7:55:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 7:55:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 7:55:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 07:55:41 dist-test-slave-vpvm krb5kdc[14549](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786262141, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.112.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:41.583024 29120 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.240s user 0.010s sys 0.138s
I20260809 07:55:41.583165 29120 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 07:55:41.583230 29120 mini_ranger_kms.cc:401] 127.28.112.20:60007/kms/v1/keys
Aug 09 07:55:41 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262141, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.112.20@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
Aug 09 07:55:42 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262141, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.112.20@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
Aug 09 07:55:44 dist-test-slave-vpvm krb5kdc[14549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:55:44.465759 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:37947
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:37947
--ranger_config_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.112.20:60007/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:55:44.574453 15240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:44.574697 15240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:44.574750 15240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:44.578668 15240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:55:44.578723 15240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:44.578749 15240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:44.578771 15240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:55:44.578799 15240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:55:44.584067 15240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.112.20:60007/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:37947
--ranger_config_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:37947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:44.585079 15240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:44.586081 15240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:44.594163 15246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:44.594177 15247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:44.594444 15240 server_base.cc:1034] running on GCE node
W20260809 07:55:44.594313 15251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:44.595127 15240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:44.596169 15240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:44.597327 15240 hybrid_clock.cc:648] HybridClock initialized: now 1786262144597302 us; error 44 us; skew 500 ppm
Aug 09 07:55:44 dist-test-slave-vpvm krb5kdc[14549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262144, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:44.603646 15240 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:55:44.604594 15240 webserver.cc:460] Webserver started at http://127.28.112.62:39223/ using document root <none> and password file <none>
I20260809 07:55:44.605105 15240 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:44.605167 15240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:44.605358 15240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 07:55:44 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262144, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
I20260809 07:55:44.664311 15240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "c01864245f9b47c3b0bf1ae9f9143b71"
format_stamp: "Formatted at 2026-08-09 07:55:44 on dist-test-slave-vpvm"
server_key: "32fbf7bb981dfaa737d335eddda59145"
server_key_iv: "186d2d84593dce973fa05adbf84dbbfc"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:44.664918 15240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "c01864245f9b47c3b0bf1ae9f9143b71"
format_stamp: "Formatted at 2026-08-09 07:55:44 on dist-test-slave-vpvm"
server_key: "32fbf7bb981dfaa737d335eddda59145"
server_key_iv: "186d2d84593dce973fa05adbf84dbbfc"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:44.668756 15240 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260809 07:55:44.686007 15262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:44.687620 15240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 07:55:44.687789 15240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "c01864245f9b47c3b0bf1ae9f9143b71"
format_stamp: "Formatted at 2026-08-09 07:55:44 on dist-test-slave-vpvm"
server_key: "32fbf7bb981dfaa737d335eddda59145"
server_key_iv: "186d2d84593dce973fa05adbf84dbbfc"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:44.687951 15240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:44.707051 15240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:44.710188 15240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:44.710378 15240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:44.718884 15240 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:37947
I20260809 07:55:44.718890 15337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:37947 every 8 connection(s)
I20260809 07:55:44.720057 15240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:55:44.722573 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 15240
I20260809 07:55:44.722708 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:55:44.723035 29120 external_mini_cluster.cc:1428] Setting key 18d1dd91b237d08d1df91fc7f78fbb6f
I20260809 07:55:44.723137 15338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:55:44.730232 15338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71: Bootstrap starting.
I20260809 07:55:44.732735 15338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71: Neither blocks nor log segments found. Creating new log.
Aug 09 07:55:44 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:55:44.733464 15338 log.cc:824] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:44.735579 15338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71: No bootstrap required, opened a new log
I20260809 07:55:44.739033 15342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:44.724481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.28.112.62:37947)
0809 07:55:44.724961 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:44.724982 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:44.725186 (+ 204us) server_negotiation.cc:407] Connection header received
0809 07:55:44.726175 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:44.726198 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:44.726628 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:44.726983 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:44.727820 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:44.729267 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:44.730448 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:44.731100 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:44.731275 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:44.734134 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:44.734141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:44.734158 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:44.734190 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:44.736204 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:44.736623 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:44.736629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:44.736640 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:44.736744 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:44.737004 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:44.737008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:44.737010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:44.737378 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:44.737584 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:44.737853 (+ 269us) server_negotiation.cc:299] Negotiation successful
0809 07:55:44.738167 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":134,"threads_started":1}
I20260809 07:55:44.739606 15338 raft_consensus.cc:348] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c01864245f9b47c3b0bf1ae9f9143b71" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37947 } }
I20260809 07:55:44.739863 15338 raft_consensus.cc:374] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:55:44.739959 15338 raft_consensus.cc:729] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c01864245f9b47c3b0bf1ae9f9143b71, State: Initialized, Role: FOLLOWER
I20260809 07:55:44.740445 15338 consensus_queue.cc:260] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c01864245f9b47c3b0bf1ae9f9143b71" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37947 } }
I20260809 07:55:44.740597 15338 raft_consensus.cc:388] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:55:44.740655 15338 raft_consensus.cc:482] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:55:44.740742 15338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:55:44.741734 15338 raft_consensus.cc:504] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c01864245f9b47c3b0bf1ae9f9143b71" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37947 } }
I20260809 07:55:44.742111 15338 leader_election.cc:302] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c01864245f9b47c3b0bf1ae9f9143b71; no voters:
I20260809 07:55:44.742478 15338 leader_election.cc:288] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:55:44.742578 15346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:55:44.742865 15346 raft_consensus.cc:686] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [term 1 LEADER]: Becoming Leader. State: Replica: c01864245f9b47c3b0bf1ae9f9143b71, State: Running, Role: LEADER
I20260809 07:55:44.743249 15346 consensus_queue.cc:237] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c01864245f9b47c3b0bf1ae9f9143b71" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37947 } }
I20260809 07:55:44.743813 15338 sys_catalog.cc:564] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:55:44.745033 15347 sys_catalog.cc:455] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c01864245f9b47c3b0bf1ae9f9143b71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c01864245f9b47c3b0bf1ae9f9143b71" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37947 } } }
I20260809 07:55:44.745067 15348 sys_catalog.cc:455] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c01864245f9b47c3b0bf1ae9f9143b71. Latest consensus state: current_term: 1 leader_uuid: "c01864245f9b47c3b0bf1ae9f9143b71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c01864245f9b47c3b0bf1ae9f9143b71" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37947 } } }
I20260809 07:55:44.745155 15347 sys_catalog.cc:458] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [sys.catalog]: This master's current role is: LEADER
I20260809 07:55:44.745182 15348 sys_catalog.cc:458] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [sys.catalog]: This master's current role is: LEADER
I20260809 07:55:44.745500 15353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:55:44.748637 15353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:55:44.749581 15338 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 07:55:44.756726 15353 catalog_manager.cc:1347] Generated new cluster ID: 6d2ae774823843ebac58026961972c65
I20260809 07:55:44.756912 15353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:55:44.787842 15353 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:55:44.789284 15353 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:55:44.796154 15353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71: Generated new TSK 0
I20260809 07:55:44.797034 15353 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
I20260809 07:55:45.559348 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.112.20:60007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:37947
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:55:45.725111 15416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:45.725343 15416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:45.725401 15416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:45.730275 15416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:45.730391 15416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:45.730505 15416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:55:45.738022 15416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.112.20:60007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:37947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:45.739168 15416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:45.740275 15416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:45.749670 15430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:45.749703 15424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:45.749876 15416 server_base.cc:1034] running on GCE node
W20260809 07:55:45.749903 15426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:45.750492 15416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:45.751084 15416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:45.752264 15416 hybrid_clock.cc:648] HybridClock initialized: now 1786262145752242 us; error 35 us; skew 500 ppm
Aug 09 07:55:45 dist-test-slave-vpvm krb5kdc[14549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262145, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:45.755134 15416 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:55:45.756013 15416 webserver.cc:460] Webserver started at http://127.28.112.1:45295/ using document root <none> and password file <none>
I20260809 07:55:45.756520 15416 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:45.756587 15416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:45.756784 15416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 07:55:45 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262145, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
I20260809 07:55:45.772802 15416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "b69d7563e79142d3a951e19acd78134c"
format_stamp: "Formatted at 2026-08-09 07:55:45 on dist-test-slave-vpvm"
server_key: "0cad802834fc96c255d984bf833d423a"
server_key_iv: "e3b612f0f72499d1a428e54ae9d1c191"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:45.773412 15416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "b69d7563e79142d3a951e19acd78134c"
format_stamp: "Formatted at 2026-08-09 07:55:45 on dist-test-slave-vpvm"
server_key: "0cad802834fc96c255d984bf833d423a"
server_key_iv: "e3b612f0f72499d1a428e54ae9d1c191"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:45.783371 15416 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.002s
I20260809 07:55:45.798872 15450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:45.800524 15416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 07:55:45.800724 15416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "b69d7563e79142d3a951e19acd78134c"
format_stamp: "Formatted at 2026-08-09 07:55:45 on dist-test-slave-vpvm"
server_key: "0cad802834fc96c255d984bf833d423a"
server_key_iv: "e3b612f0f72499d1a428e54ae9d1c191"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:45.800930 15416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:45.825995 15416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:45.828769 15416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:45.828951 15416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:45.829564 15416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:55:45.830475 15416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:55:45.830526 15416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:45.830566 15416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:55:45.830592 15416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:45.840610 15416 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:33497
I20260809 07:55:45.840873 15633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:33497 every 8 connection(s)
I20260809 07:55:45.841806 15416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:55:45.848207 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 15416
I20260809 07:55:45.848335 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:55:45.848614 29120 external_mini_cluster.cc:1428] Setting key 2687aa021ed6bce87ff3ae95a9176810
Aug 09 07:55:45 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262145, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:55:45.865203 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:45.844417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37947 (local address 127.28.112.1:54017)
0809 07:55:45.846288 (+ 1871us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:45.846346 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:45.847517 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:45.848540 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:45.848550 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:45.848929 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:45.849828 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:45.849850 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:45.851382 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:45.851386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:45.852782 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:45.852797 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:45.853486 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:45.853491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:45.853692 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:45.854378 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:45.854451 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:45.856207 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:45.862060 (+ 5853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:45.862067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:45.862084 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:45.862413 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:45.862737 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:45.862740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:45.862742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:45.862909 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:45.863394 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:45.863400 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:45.863717 (+ 317us) client_negotiation.cc:769] Sending connection context
0809 07:55:45.863903 (+ 186us) client_negotiation.cc:240] Negotiation successful
0809 07:55:45.864208 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1636,"thread_start_us":138,"threads_started":1}
I20260809 07:55:45.865935 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:45.843975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:54017 (local address 127.28.112.62:37947)
0809 07:55:45.847973 (+ 3998us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:45.847979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:45.847997 (+ 18us) server_negotiation.cc:407] Connection header received
0809 07:55:45.848048 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:45.848052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:45.848106 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:45.848206 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:45.850359 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:45.851202 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:45.852948 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:45.853321 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:45.853430 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:45.856389 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:45.856393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:45.856396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:45.856423 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:45.858210 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:45.862544 (+ 4334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:45.862548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:45.862550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:45.862640 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:45.863010 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:45.863014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:45.863015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:45.863247 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:45.863337 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:45.865648 (+ 2311us) server_negotiation.cc:299] Negotiation successful
0809 07:55:45.865775 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3905,"thread_start_us":90,"threads_started":1}
I20260809 07:55:45.867722 15636 heartbeater.cc:347] Connected to a master server at 127.28.112.62:37947
I20260809 07:55:45.867965 15636 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:45.868534 15636 heartbeater.cc:511] Master 127.28.112.62:37947 requested a full tablet report, sending...
I20260809 07:55:45.870381 15286 ts_manager.cc:194] Registered new tserver with Master: b69d7563e79142d3a951e19acd78134c (127.28.112.1:33497)
I20260809 07:55:45.871899 15286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:54017
WARNING: no policy specified for kudu/127.28.112.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
I20260809 07:55:45.894651 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.112.20:60007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:37947
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 07:55:45 dist-test-slave-vpvm krb5kdc[14549](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786262145, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 07:55:46.054284 15650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:46.054606 15650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:46.054672 15650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:46.060461 15650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:46.060586 15650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:46.060700 15650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:55:46.068291 15650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.112.20:60007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:37947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:46.069635 15650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:46.071049 15650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:46.082253 15664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:46.082358 15659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:46.082257 15662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:46.083014 15650 server_base.cc:1034] running on GCE node
I20260809 07:55:46.083505 15650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:46.084309 15650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:46.085659 15650 hybrid_clock.cc:648] HybridClock initialized: now 1786262146085586 us; error 85 us; skew 500 ppm
Aug 09 07:55:46 dist-test-slave-vpvm krb5kdc[14549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262146, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:46.089370 15650 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:55:46.090451 15650 webserver.cc:460] Webserver started at http://127.28.112.2:42865/ using document root <none> and password file <none>
I20260809 07:55:46.091185 15650 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:46.091272 15650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:46.091497 15650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 07:55:46 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262146, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
I20260809 07:55:46.108121 15650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "2c8d0183d9f14b9b9b17d135cbd88e42"
format_stamp: "Formatted at 2026-08-09 07:55:46 on dist-test-slave-vpvm"
server_key: "ea054f5e38878def2ab49b5b93b8ef14"
server_key_iv: "9a1a9162ae910b73710fcfca76d39ab0"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:46.108893 15650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "2c8d0183d9f14b9b9b17d135cbd88e42"
format_stamp: "Formatted at 2026-08-09 07:55:46 on dist-test-slave-vpvm"
server_key: "ea054f5e38878def2ab49b5b93b8ef14"
server_key_iv: "9a1a9162ae910b73710fcfca76d39ab0"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:46.113161 15650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 07:55:46.129499 15671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:46.130947 15650 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 07:55:46.131068 15650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "2c8d0183d9f14b9b9b17d135cbd88e42"
format_stamp: "Formatted at 2026-08-09 07:55:46 on dist-test-slave-vpvm"
server_key: "ea054f5e38878def2ab49b5b93b8ef14"
server_key_iv: "9a1a9162ae910b73710fcfca76d39ab0"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:46.131187 15650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:46.158740 15650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:46.162250 15650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:46.162468 15650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:46.163237 15650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:55:46.164341 15650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:55:46.164407 15650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:46.164481 15650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:55:46.164512 15650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:46.175207 15650 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:34949
I20260809 07:55:46.176393 15650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:55:46.183691 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 15650
I20260809 07:55:46.183840 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:55:46.184145 29120 external_mini_cluster.cc:1428] Setting key c02f657412ada7c5009eb171b992c53e
I20260809 07:55:46.190433 15861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:34949 every 8 connection(s)
WARNING: no policy specified for kudu/127.28.112.3@KRBTEST.COM; defaulting to no policy
Aug 09 07:55:46 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262146, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.3@KRBTEST.COM" created.
I20260809 07:55:46.222493 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:46.194528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:33261 (local address 127.28.112.62:37947)
0809 07:55:46.194662 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:46.194667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:46.200496 (+ 5829us) server_negotiation.cc:407] Connection header received
0809 07:55:46.205519 (+ 5023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:46.205524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:46.205589 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:46.205652 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:46.207698 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:46.208529 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:46.211975 (+ 3446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:46.212324 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:46.212453 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:46.217176 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:46.217180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:46.217183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:46.217213 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:46.219364 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:46.220004 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:46.220008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:46.220010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:46.220102 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:46.220520 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:46.220523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:46.220525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:46.220762 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:46.220847 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:46.222144 (+ 1297us) server_negotiation.cc:299] Negotiation successful
0809 07:55:46.222295 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 07:55:46.231999 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:46.193605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37947 (local address 127.28.112.2:33261)
0809 07:55:46.200317 (+ 6712us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:46.200382 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:46.201539 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:46.206222 (+ 4683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:46.206233 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:46.206664 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:46.207501 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:46.207524 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:46.208704 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:46.208707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:46.210227 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:46.210239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:46.212689 (+ 2450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:46.212693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:46.212911 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:46.213883 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:46.213905 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:46.216948 (+ 3043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:46.219516 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:46.219522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:46.219542 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:46.219884 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:46.220208 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:46.220210 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:46.220212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:46.220396 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:46.220919 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:46.220926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:46.221254 (+ 328us) client_negotiation.cc:769] Sending connection context
0809 07:55:46.222762 (+ 1508us) client_negotiation.cc:240] Negotiation successful
0809 07:55:46.223173 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6456,"thread_start_us":152,"threads_started":1}
I20260809 07:55:46.233646 15862 heartbeater.cc:347] Connected to a master server at 127.28.112.62:37947
I20260809 07:55:46.233942 15862 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:46.234699 15862 heartbeater.cc:511] Master 127.28.112.62:37947 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
I20260809 07:55:46.237232 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.112.20:60007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:37947
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 07:55:46.238481 15286 ts_manager.cc:194] Registered new tserver with Master: 2c8d0183d9f14b9b9b17d135cbd88e42 (127.28.112.2:34949)
I20260809 07:55:46.239292 15286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:33261
W20260809 07:55:46.407171 15873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:46.407673 15873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:46.407785 15873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:46.413550 15873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:46.413724 15873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:46.413851 15873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:55:46.421111 15873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.112.20:60007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:37947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:46.422524 15873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:46.423806 15873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:46.436215 15881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:46.436709 15879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:46.436947 15884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:46.438050 15873 server_base.cc:1034] running on GCE node
I20260809 07:55:46.438591 15873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:46.439291 15873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:46.441221 15873 hybrid_clock.cc:648] HybridClock initialized: now 1786262146441159 us; error 73 us; skew 500 ppm
Aug 09 07:55:46 dist-test-slave-vpvm krb5kdc[14549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262146, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:46.444937 15873 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:55:46.446123 15873 webserver.cc:460] Webserver started at http://127.28.112.3:36331/ using document root <none> and password file <none>
I20260809 07:55:46.446920 15873 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:46.447057 15873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:46.447317 15873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 07:55:46 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262146, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
I20260809 07:55:46.466248 15873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "29f1d457bdef499ca9946e83387f5ed1"
format_stamp: "Formatted at 2026-08-09 07:55:46 on dist-test-slave-vpvm"
server_key: "0da76c931e93df9d148083e7d497cdc3"
server_key_iv: "75a2d65e154af02340fa6a7bb11dedf5"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:46.467903 15873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "29f1d457bdef499ca9946e83387f5ed1"
format_stamp: "Formatted at 2026-08-09 07:55:46 on dist-test-slave-vpvm"
server_key: "0da76c931e93df9d148083e7d497cdc3"
server_key_iv: "75a2d65e154af02340fa6a7bb11dedf5"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:46.478552 15873 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.000s
I20260809 07:55:46.501279 15893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:46.503095 15873 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260809 07:55:46.503265 15873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "29f1d457bdef499ca9946e83387f5ed1"
format_stamp: "Formatted at 2026-08-09 07:55:46 on dist-test-slave-vpvm"
server_key: "0da76c931e93df9d148083e7d497cdc3"
server_key_iv: "75a2d65e154af02340fa6a7bb11dedf5"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:46.503437 15873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 09 07:55:46 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262145, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
I20260809 07:55:46.522181 15873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:46.525614 15873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:46.525818 15873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:46.526547 15873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:55:46.527944 15873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:55:46.528007 15873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:46.528067 15873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:55:46.528095 15873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:46.539944 15873 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:37343
I20260809 07:55:46.541250 15873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:55:46.543870 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 15873
I20260809 07:55:46.544021 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:55:46.544304 29120 external_mini_cluster.cc:1428] Setting key 278d46b934b9f5b73eaaa9cdfebde7e9
I20260809 07:55:46.550836 16070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:37343 every 8 connection(s)
Aug 09 07:55:46 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262146, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:55:46.592613 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:46.561503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:40909 (local address 127.28.112.62:37947)
0809 07:55:46.561628 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:46.561633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:46.570740 (+ 9107us) server_negotiation.cc:407] Connection header received
0809 07:55:46.572034 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:46.572037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:46.572092 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:46.572140 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:46.573918 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:46.574774 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:46.581087 (+ 6313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:46.581444 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:46.581525 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:46.588468 (+ 6943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:46.588472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:46.588475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:46.588506 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:46.590176 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:46.590806 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:46.590809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:46.590811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:46.590898 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:46.591316 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:46.591319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:46.591321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:46.591548 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:46.591626 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:46.592316 (+ 690us) server_negotiation.cc:299] Negotiation successful
0809 07:55:46.592442 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 07:55:46.594575 16077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:46.560361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37947 (local address 127.28.112.3:40909)
0809 07:55:46.570562 (+ 10201us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:46.570640 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:46.571765 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:46.572597 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:46.572607 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:46.573023 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:46.573749 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:46.573769 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:46.574940 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:46.574943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:46.576347 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:46.576358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:46.581754 (+ 5396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:46.581758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:46.582019 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:46.586406 (+ 4387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:46.586429 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:46.588253 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:46.590323 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:46.590330 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:46.590349 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:46.590690 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:46.591014 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:46.591017 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:46.591019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:46.591203 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:46.591695 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:46.591701 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:46.592035 (+ 334us) client_negotiation.cc:769] Sending connection context
0809 07:55:46.593130 (+ 1095us) client_negotiation.cc:240] Negotiation successful
0809 07:55:46.593489 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9903,"thread_start_us":182,"threads_started":1}
I20260809 07:55:46.596439 16071 heartbeater.cc:347] Connected to a master server at 127.28.112.62:37947
I20260809 07:55:46.596683 16071 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:46.597239 16071 heartbeater.cc:511] Master 127.28.112.62:37947 requested a full tablet report, sending...
I20260809 07:55:46.598448 15286 ts_manager.cc:194] Registered new tserver with Master: 29f1d457bdef499ca9946e83387f5ed1 (127.28.112.3:37343)
I20260809 07:55:46.599257 15286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:40909
I20260809 07:55:46.605702 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 07:55:46.606462 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 15416
I20260809 07:55:46.653466 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 15650
I20260809 07:55:46.664798 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 15873
I20260809 07:55:46.679534 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 15240
I20260809 07:55:46.694955 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:37947
--webserver_interface=127.28.112.62
--webserver_port=39223
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:37947
--ranger_config_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.112.20:60007/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.112.20:60007/kms with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:55:46.802388 16078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:46.802620 16078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:46.802665 16078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:46.806526 16078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:55:46.806576 16078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:46.806594 16078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:46.806609 16078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:55:46.806623 16078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:55:46.811801 16078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.112.20:60007/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:37947
--ranger_config_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:37947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=39223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:46.812700 16078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:46.813606 16078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:46.821537 16084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:46.821655 16085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:46.821761 16078 server_base.cc:1034] running on GCE node
W20260809 07:55:46.821528 16090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:46.822348 16078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:46.823385 16078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:46.824534 16078 hybrid_clock.cc:648] HybridClock initialized: now 1786262146824512 us; error 40 us; skew 500 ppm
Aug 09 07:55:46 dist-test-slave-vpvm krb5kdc[14549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262146, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:46.827121 16078 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:55:46.827947 16078 webserver.cc:460] Webserver started at http://127.28.112.62:39223/ using document root <none> and password file <none>
I20260809 07:55:46.828428 16078 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:46.828501 16078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 07:55:46 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262146, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
I20260809 07:55:46.900468 16078 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 07:55:46.902837 16102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:46.904213 16078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 07:55:46.904338 16078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "c01864245f9b47c3b0bf1ae9f9143b71"
format_stamp: "Formatted at 2026-08-09 07:55:44 on dist-test-slave-vpvm"
server_key: "32fbf7bb981dfaa737d335eddda59145"
server_key_iv: "186d2d84593dce973fa05adbf84dbbfc"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:46.904886 16078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:46.922735 16078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:46.925415 16078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:46.925595 16078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:46.932776 16078 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:37947
I20260809 07:55:46.932814 16192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:37947 every 8 connection(s)
I20260809 07:55:46.933821 16078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:55:46.938201 16193 sys_catalog.cc:263] Verifying existing consensus state
I20260809 07:55:46.939888 16193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71: Bootstrap starting.
I20260809 07:55:46.942312 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 16078
I20260809 07:55:46.942741 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.112.20:60007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:33497
--local_ip_for_outbound_sockets=127.28.112.1
--tserver_master_addrs=127.28.112.62:37947
--webserver_port=45295
--webserver_interface=127.28.112.1
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.112.20:60007/kms with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 07:55:46.957178 16193 log.cc:824] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:46.963593 16193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 07:55:46.963943 16193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71: Bootstrap complete.
I20260809 07:55:46.968050 16193 raft_consensus.cc:348] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c01864245f9b47c3b0bf1ae9f9143b71" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37947 } }
I20260809 07:55:46.968425 16193 raft_consensus.cc:729] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c01864245f9b47c3b0bf1ae9f9143b71, State: Initialized, Role: FOLLOWER
I20260809 07:55:46.968906 16193 consensus_queue.cc:260] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c01864245f9b47c3b0bf1ae9f9143b71" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37947 } }
I20260809 07:55:46.969040 16193 raft_consensus.cc:388] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:55:46.969107 16193 raft_consensus.cc:482] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:55:46.969201 16193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [term 1 FOLLOWER]: Advancing to term 2
I20260809 07:55:46.971067 16193 raft_consensus.cc:504] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c01864245f9b47c3b0bf1ae9f9143b71" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37947 } }
I20260809 07:55:46.971374 16193 leader_election.cc:302] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c01864245f9b47c3b0bf1ae9f9143b71; no voters:
I20260809 07:55:46.971655 16193 leader_election.cc:288] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 07:55:46.971787 16198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 07:55:46.972039 16198 raft_consensus.cc:686] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [term 2 LEADER]: Becoming Leader. State: Replica: c01864245f9b47c3b0bf1ae9f9143b71, State: Running, Role: LEADER
I20260809 07:55:46.972467 16198 consensus_queue.cc:237] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c01864245f9b47c3b0bf1ae9f9143b71" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37947 } }
I20260809 07:55:46.972828 16193 sys_catalog.cc:564] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:55:46.973978 16200 sys_catalog.cc:455] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c01864245f9b47c3b0bf1ae9f9143b71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c01864245f9b47c3b0bf1ae9f9143b71" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37947 } } }
I20260809 07:55:46.974114 16200 sys_catalog.cc:458] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [sys.catalog]: This master's current role is: LEADER
I20260809 07:55:46.974488 16202 sys_catalog.cc:455] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c01864245f9b47c3b0bf1ae9f9143b71. Latest consensus state: current_term: 2 leader_uuid: "c01864245f9b47c3b0bf1ae9f9143b71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c01864245f9b47c3b0bf1ae9f9143b71" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37947 } } }
I20260809 07:55:46.974627 16202 sys_catalog.cc:458] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71 [sys.catalog]: This master's current role is: LEADER
I20260809 07:55:46.974920 16215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:55:46.975102 16193 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 07:55:46.981052 16215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:55:46.981935 16215 catalog_manager.cc:1259] Loaded cluster ID: 6d2ae774823843ebac58026961972c65
I20260809 07:55:46.982056 16215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:55:46.985553 16215 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:55:46.986416 16215 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71: Loaded TSK: 0
I20260809 07:55:46.987812 16215 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 07:55:47.081243 16195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:47.081490 16195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:47.081550 16195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:47.085569 16195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:47.085631 16195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:47.085711 16195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:55:47.092862 16195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.112.20:60007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:33497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=45295
--tserver_master_addrs=127.28.112.62:37947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:47.094348 16195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:47.095547 16195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:47.105285 16263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:47.105285 16266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:47.105705 16264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:47.106129 16195 server_base.cc:1034] running on GCE node
I20260809 07:55:47.106586 16195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:47.107254 16195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:47.111058 16195 hybrid_clock.cc:648] HybridClock initialized: now 1786262147111019 us; error 47 us; skew 500 ppm
Aug 09 07:55:47 dist-test-slave-vpvm krb5kdc[14549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:47.114786 16195 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:55:47.115942 16195 webserver.cc:460] Webserver started at http://127.28.112.1:45295/ using document root <none> and password file <none>
I20260809 07:55:47.116634 16195 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:47.116729 16195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 07:55:47 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
I20260809 07:55:47.141908 16195 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260809 07:55:47.145291 16278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:47.146894 16195 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260809 07:55:47.147099 16195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "b69d7563e79142d3a951e19acd78134c"
format_stamp: "Formatted at 2026-08-09 07:55:45 on dist-test-slave-vpvm"
server_key: "0cad802834fc96c255d984bf833d423a"
server_key_iv: "e3b612f0f72499d1a428e54ae9d1c191"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:47.147859 16195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:47.159104 16195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:47.161903 16195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:47.162117 16195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:47.162746 16195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:55:47.163601 16195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:55:47.163650 16195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:47.163702 16195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:55:47.163730 16195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:47.174093 16195 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:33497
I20260809 07:55:47.175474 16195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:55:47.176384 16464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:33497 every 8 connection(s)
I20260809 07:55:47.180835 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 16195
I20260809 07:55:47.181355 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.112.20:60007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:34949
--local_ip_for_outbound_sockets=127.28.112.2
--tserver_master_addrs=127.28.112.62:37947
--webserver_port=42865
--webserver_interface=127.28.112.2
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.112.20:60007/kms with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 07:55:47 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:55:47.195420 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:47.178471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:48823 (local address 127.28.112.62:37947)
0809 07:55:47.180061 (+ 1590us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:47.180082 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:47.180129 (+ 47us) server_negotiation.cc:407] Connection header received
0809 07:55:47.180662 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:47.180677 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:47.181076 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:47.181456 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:47.183157 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:47.184227 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:47.185791 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:47.186565 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:47.186751 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:47.189436 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:47.189443 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:47.189460 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:47.189493 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:47.191472 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:47.192122 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:47.192128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:47.192134 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:47.192280 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:47.192712 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:47.192716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:47.192718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:47.193185 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:47.193291 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:47.194070 (+ 779us) server_negotiation.cc:299] Negotiation successful
0809 07:55:47.194375 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1428,"thread_start_us":118,"threads_started":1}
I20260809 07:55:47.197404 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:47.178686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37947 (local address 127.28.112.1:48823)
0809 07:55:47.179166 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:47.179218 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:47.180220 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:47.181944 (+ 1724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:47.181985 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:47.182313 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:47.182932 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:47.182955 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:47.184367 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:47.184371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:47.185643 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:47.185658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:47.186725 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:47.186728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:47.186867 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:47.187459 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:47.187481 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:47.189208 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:47.191610 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:47.191617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:47.191630 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:47.191978 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:47.192408 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:47.192410 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:47.192412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:47.192588 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:47.193383 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:47.193389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:47.193677 (+ 288us) client_negotiation.cc:769] Sending connection context
0809 07:55:47.196179 (+ 2502us) client_negotiation.cc:240] Negotiation successful
0809 07:55:47.196490 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":139,"threads_started":1}
I20260809 07:55:47.201963 16465 heartbeater.cc:347] Connected to a master server at 127.28.112.62:37947
I20260809 07:55:47.202224 16465 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:47.202795 16465 heartbeater.cc:511] Master 127.28.112.62:37947 requested a full tablet report, sending...
W20260809 07:55:47.206255 16465 heartbeater.cc:644] Failed to heartbeat to 127.28.112.62:37947 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 07:55:47.206387 16465 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:47.206588 16465 heartbeater.cc:511] Master 127.28.112.62:37947 requested a full tablet report, sending...
I20260809 07:55:47.207644 16465 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:47.207821 16465 heartbeater.cc:511] Master 127.28.112.62:37947 requested a full tablet report, sending...
W20260809 07:55:47.213179 16465 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 07:55:47.348198 16470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:47.348640 16470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:47.348755 16470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:47.354692 16470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:47.354815 16470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:47.354966 16470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:55:47.360893 16470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.112.20:60007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:34949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=42865
--tserver_master_addrs=127.28.112.62:37947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:47.362027 16470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:47.363101 16470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:47.374897 16478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:47.374883 16479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:47.374883 16483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:47.375232 16470 server_base.cc:1034] running on GCE node
I20260809 07:55:47.375631 16470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:47.376246 16470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:47.377408 16470 hybrid_clock.cc:648] HybridClock initialized: now 1786262147377388 us; error 29 us; skew 500 ppm
Aug 09 07:55:47 dist-test-slave-vpvm krb5kdc[14549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:47.380587 16470 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:55:47.381490 16470 webserver.cc:460] Webserver started at http://127.28.112.2:42865/ using document root <none> and password file <none>
I20260809 07:55:47.382093 16470 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:47.382182 16470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 07:55:47 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
I20260809 07:55:47.404196 16470 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260809 07:55:47.407678 16498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:47.408974 16470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 07:55:47.409090 16470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "2c8d0183d9f14b9b9b17d135cbd88e42"
format_stamp: "Formatted at 2026-08-09 07:55:46 on dist-test-slave-vpvm"
server_key: "ea054f5e38878def2ab49b5b93b8ef14"
server_key_iv: "9a1a9162ae910b73710fcfca76d39ab0"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:47.409807 16470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:47.428155 16470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:47.431027 16470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:47.431200 16470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:47.431788 16470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:55:47.432667 16470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:55:47.432725 16470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:47.432804 16470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:55:47.432830 16470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:47.443027 16470 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:34949
I20260809 07:55:47.444414 16470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:55:47.448014 16680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:34949 every 8 connection(s)
I20260809 07:55:47.452944 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 16470
I20260809 07:55:47.453464 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.112.20:60007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:37343
--local_ip_for_outbound_sockets=127.28.112.3
--tserver_master_addrs=127.28.112.62:37947
--webserver_port=36331
--webserver_interface=127.28.112.3
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.112.20:60007/kms with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 07:55:47 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:55:47.464855 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:47.450165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:54111 (local address 127.28.112.62:37947)
0809 07:55:47.450305 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:47.450310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:47.452309 (+ 1999us) server_negotiation.cc:407] Connection header received
0809 07:55:47.453788 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:47.453792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:47.453853 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:47.453907 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:47.455571 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:47.456384 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:47.458062 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:47.458446 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:47.458526 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:47.461047 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:47.461050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:47.461052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:47.461075 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:47.462379 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:47.462974 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:47.462977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:47.462979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:47.463072 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:47.463527 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:47.463531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:47.463533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:47.463763 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:47.463852 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:47.464551 (+ 699us) server_negotiation.cc:299] Negotiation successful
0809 07:55:47.464685 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 07:55:47.466708 16684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:47.450254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37947 (local address 127.28.112.2:54111)
0809 07:55:47.452163 (+ 1909us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:47.452206 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:47.453442 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:47.454376 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:47.454386 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:47.454739 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:47.455371 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:47.455390 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:47.456571 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:47.456575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:47.457896 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:47.457909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:47.458869 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:47.458872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:47.459025 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:47.459571 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:47.459588 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:47.460874 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:47.462493 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:47.462499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:47.462514 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:47.462846 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:47.463208 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:47.463211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:47.463214 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:47.463394 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:47.463928 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:47.463935 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:47.464239 (+ 304us) client_negotiation.cc:769] Sending connection context
0809 07:55:47.465310 (+ 1071us) client_negotiation.cc:240] Negotiation successful
0809 07:55:47.465651 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1716,"thread_start_us":143,"threads_started":1}
I20260809 07:55:47.468261 16681 heartbeater.cc:347] Connected to a master server at 127.28.112.62:37947
I20260809 07:55:47.468515 16681 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:47.469084 16681 heartbeater.cc:511] Master 127.28.112.62:37947 requested a full tablet report, sending...
W20260809 07:55:47.470825 16681 heartbeater.cc:644] Failed to heartbeat to 127.28.112.62:37947 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 07:55:47.470948 16681 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:47.471112 16681 heartbeater.cc:511] Master 127.28.112.62:37947 requested a full tablet report, sending...
I20260809 07:55:47.472018 16681 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:47.472174 16681 heartbeater.cc:511] Master 127.28.112.62:37947 requested a full tablet report, sending...
W20260809 07:55:47.473044 16681 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 07:55:47.615474 16686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:47.615813 16686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:47.615896 16686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:47.621706 16686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:47.621778 16686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:47.621863 16686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:55:47.629273 16686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.112.20:60007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:37343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=36331
--tserver_master_addrs=127.28.112.62:37947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:47.630616 16686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:47.631929 16686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:47.643774 16695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:47.643779 16697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:47.644341 16702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:47.644866 16686 server_base.cc:1034] running on GCE node
I20260809 07:55:47.645737 16686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:47.646562 16686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:47.647742 16686 hybrid_clock.cc:648] HybridClock initialized: now 1786262147647712 us; error 44 us; skew 500 ppm
Aug 09 07:55:47 dist-test-slave-vpvm krb5kdc[14549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:47.651273 16686 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:55:47.652453 16686 webserver.cc:460] Webserver started at http://127.28.112.3:36331/ using document root <none> and password file <none>
I20260809 07:55:47.653246 16686 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:47.653405 16686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 07:55:47 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
I20260809 07:55:47.674576 16686 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 07:55:47.678290 16713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:47.679765 16686 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 07:55:47.679888 16686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "29f1d457bdef499ca9946e83387f5ed1"
format_stamp: "Formatted at 2026-08-09 07:55:46 on dist-test-slave-vpvm"
server_key: "0da76c931e93df9d148083e7d497cdc3"
server_key_iv: "75a2d65e154af02340fa6a7bb11dedf5"
server_key_version: "kuduclusterkey@0"
I20260809 07:55:47.680572 16686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:47.697575 16686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:47.700757 16686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:47.700909 16686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:47.701452 16686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:55:47.702399 16686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:55:47.702455 16686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:47.702513 16686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:55:47.702545 16686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:47.712677 16686 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:37343
I20260809 07:55:47.712718 16897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:37343 every 8 connection(s)
I20260809 07:55:47.713629 16686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:55:47.716382 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 16686
I20260809 07:55:47.726831 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:47.717130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59338 (local address 127.28.112.62:37947)
0809 07:55:47.717261 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:47.717266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:47.717546 (+ 280us) server_negotiation.cc:407] Connection header received
0809 07:55:47.717766 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:47.717770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:47.717824 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:47.717875 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:47.718952 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:47.719681 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:47.720870 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:47.721196 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:47.721265 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:47.723073 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:47.723077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:47.723080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:47.723106 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:47.724698 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:47.725167 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:47.725171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:47.725173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:47.725250 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:47.725535 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:47.725538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:47.725540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:47.725754 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:47.725917 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:47.726565 (+ 648us) server_negotiation.cc:299] Negotiation successful
0809 07:55:47.726673 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Aug 09 07:55:47 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:55:47.732689 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:47.718108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:38657 (local address 127.28.112.62:37947)
0809 07:55:47.719595 (+ 1487us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:47.719599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:47.721105 (+ 1506us) server_negotiation.cc:407] Connection header received
0809 07:55:47.721140 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:47.721142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:47.721189 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:47.721264 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:47.722700 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:47.723441 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:47.725069 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:47.725393 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:47.725494 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:47.728282 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:47.728286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:47.728288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:47.728311 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:47.730362 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:47.730969 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:47.730973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:47.730974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:47.731063 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:47.731474 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:47.731478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:47.731480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:47.731690 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:47.731789 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:47.732397 (+ 608us) server_negotiation.cc:299] Negotiation successful
0809 07:55:47.732524 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":1413,"spinlock_wait_cycles":51456,"thread_start_us":83,"threads_started":1}
I20260809 07:55:47.733718 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:47.718581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37947 (local address 127.28.112.3:38657)
0809 07:55:47.719848 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:47.719903 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:47.720858 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:47.721507 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:47.721516 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:47.721840 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:47.722528 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:47.722550 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:47.723584 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:47.723587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:47.724938 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:47.724949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:47.725683 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:47.725688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:47.725871 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:47.726376 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:47.726396 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:47.728118 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:47.730484 (+ 2366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:47.730491 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:47.730505 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:47.730851 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:47.731160 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:47.731163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:47.731165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:47.731348 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:47.731797 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:47.731804 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:47.732111 (+ 307us) client_negotiation.cc:769] Sending connection context
0809 07:55:47.732316 (+ 205us) client_negotiation.cc:240] Negotiation successful
0809 07:55:47.732628 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1051,"thread_start_us":137,"threads_started":1}
I20260809 07:55:47.734880 16898 heartbeater.cc:347] Connected to a master server at 127.28.112.62:37947
I20260809 07:55:47.735139 16898 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:47.735706 16898 heartbeater.cc:511] Master 127.28.112.62:37947 requested a full tablet report, sending...
W20260809 07:55:47.737206 16898 heartbeater.cc:644] Failed to heartbeat to 127.28.112.62:37947 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 07:55:47.737315 16898 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:47.737475 16898 heartbeater.cc:511] Master 127.28.112.62:37947 requested a full tablet report, sending...
I20260809 07:55:47.738215 16898 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:47.738372 16898 heartbeater.cc:511] Master 127.28.112.62:37947 requested a full tablet report, sending...
W20260809 07:55:47.739127 16898 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 09 07:55:48 dist-test-slave-vpvm krb5kdc[14549](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786262148, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:48.213451 16465 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:48.213793 16465 heartbeater.cc:511] Master 127.28.112.62:37947 requested a full tablet report, sending...
I20260809 07:55:48.215158 16137 ts_manager.cc:194] Registered new tserver with Master: b69d7563e79142d3a951e19acd78134c (127.28.112.1:33497)
I20260809 07:55:48.217129 16137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:48823
I20260809 07:55:48.473280 16681 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:48.473590 16681 heartbeater.cc:511] Master 127.28.112.62:37947 requested a full tablet report, sending...
I20260809 07:55:48.474488 16136 ts_manager.cc:194] Registered new tserver with Master: 2c8d0183d9f14b9b9b17d135cbd88e42 (127.28.112.2:34949)
I20260809 07:55:48.475318 16136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:54111
Aug 09 07:55:48 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262148, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
I20260809 07:55:48.739355 16898 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:48.739662 16898 heartbeater.cc:511] Master 127.28.112.62:37947 requested a full tablet report, sending...
I20260809 07:55:48.740450 16136 ts_manager.cc:194] Registered new tserver with Master: 29f1d457bdef499ca9946e83387f5ed1 (127.28.112.3:37343)
I20260809 07:55:48.741245 16136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:38657
I20260809 07:55:48.749608 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 07:55:48.761559 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:48.751325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42626 (local address 127.28.112.62:37947)
0809 07:55:48.751615 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:48.751619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:48.751690 (+ 71us) server_negotiation.cc:407] Connection header received
0809 07:55:48.751828 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:48.751832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:48.751881 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:48.751975 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:48.752821 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.753650 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.754872 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.755244 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.755342 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:48.757097 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:48.757101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:48.757104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:48.757132 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:48.759307 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:48.759823 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:48.759827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:48.759828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:48.759920 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:48.760223 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:48.760227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:48.760229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:48.760450 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:48.760591 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:48.761245 (+ 654us) server_negotiation.cc:299] Negotiation successful
0809 07:55:48.761369 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":104,"threads_started":1}
I20260809 07:55:48.765635 16136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:55:48.770362 16136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 09 07:55:48 dist-test-slave-vpvm krb5kdc[14549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262148, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for HTTP/127.28.112.20@KRBTEST.COM
I20260809 07:55:48.803321 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:48.793490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:37343 (local address 127.0.0.1:55310)
0809 07:55:48.794581 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:48.794593 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:48.794672 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:48.795397 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:48.795404 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:48.795430 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:55:48.795601 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:48.795609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.797863 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.797867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:48.800785 (+ 2918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:48.800798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.801935 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.801939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:48.802076 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:48.802896 (+ 820us) client_negotiation.cc:769] Sending connection context
0809 07:55:48.802990 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 07:55:48.803133 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"mutex_wait_us":22,"thread_start_us":90,"threads_started":1}
I20260809 07:55:48.804206 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:48.793725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55310 (local address 127.28.112.3:37343)
0809 07:55:48.794906 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:48.794914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:48.794938 (+ 24us) server_negotiation.cc:407] Connection header received
0809 07:55:48.795016 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:48.795022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:48.795223 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:48.795770 (+ 547us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:55:48.795927 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.797586 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.800949 (+ 3363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.801793 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.803726 (+ 1933us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:48.803898 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:48.803957 (+ 59us) server_negotiation.cc:299] Negotiation successful
0809 07:55:48.804060 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1070,"thread_start_us":77,"threads_started":1}
I20260809 07:55:48.808437 16785 tablet_service.cc:1467] Processing CreateTablet for tablet 364d7d86555a4878a78a53c31b2db9e8 (DEFAULT_TABLE table=test-table [id=7716094a3de8448e89e315f277429cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:55:48.809650 16785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 364d7d86555a4878a78a53c31b2db9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:55:48.817859 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:48.791975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:34949 (local address 127.0.0.1:52492)
0809 07:55:48.794301 (+ 2326us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:48.794314 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:48.794394 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:48.814084 (+ 19690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:48.814087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:48.814110 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:55:48.814217 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:48.814224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.815449 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.815451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:48.816481 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:48.816493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.817432 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.817436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:48.817523 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:48.817548 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 07:55:48.817641 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 07:55:48.817712 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2266,"thread_start_us":80,"threads_started":1}
I20260809 07:55:48.817899 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:48.792757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52492 (local address 127.28.112.2:34949)
0809 07:55:48.796641 (+ 3884us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:48.796648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:48.796671 (+ 23us) server_negotiation.cc:407] Connection header received
0809 07:55:48.797226 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:48.797233 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:48.813903 (+ 16670us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:48.814109 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:55:48.814353 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.815332 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.816618 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.817311 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.817413 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:48.817565 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:48.817673 (+ 108us) server_negotiation.cc:299] Negotiation successful
0809 07:55:48.817781 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3770,"thread_start_us":75,"threads_started":1}
I20260809 07:55:48.818926 16953 tablet_bootstrap.cc:492] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1: Bootstrap starting.
I20260809 07:55:48.821297 16953 tablet_bootstrap.cc:654] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1: Neither blocks nor log segments found. Creating new log.
I20260809 07:55:48.821130 16574 tablet_service.cc:1467] Processing CreateTablet for tablet 364d7d86555a4878a78a53c31b2db9e8 (DEFAULT_TABLE table=test-table [id=7716094a3de8448e89e315f277429cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:55:48.822268 16574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 364d7d86555a4878a78a53c31b2db9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:55:48.822778 16953 log.cc:824] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:48.827524 16945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:48.791896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:33497 (local address 127.0.0.1:44322)
0809 07:55:48.793877 (+ 1981us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:48.793925 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:48.794079 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:48.822613 (+ 28534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:48.822617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:48.822640 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:55:48.822740 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:48.822746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.824920 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.824924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:48.826037 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:48.826049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.827123 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.827127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:48.827235 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:48.827265 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 07:55:48.827318 (+ 53us) client_negotiation.cc:240] Negotiation successful
0809 07:55:48.827383 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1605,"thread_start_us":114,"threads_started":1}
I20260809 07:55:48.828488 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:48.792758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44322 (local address 127.28.112.1:33497)
0809 07:55:48.821812 (+ 29054us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:48.821820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:48.821846 (+ 26us) server_negotiation.cc:407] Connection header received
0809 07:55:48.821925 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:48.821930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:48.822158 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:48.822894 (+ 736us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:55:48.823039 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.824794 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.826177 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.826979 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.828052 (+ 1073us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:48.828208 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:48.828260 (+ 52us) server_negotiation.cc:299] Negotiation successful
0809 07:55:48.828350 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28928,"thread_start_us":85,"threads_started":1}
I20260809 07:55:48.830886 16958 tablet_bootstrap.cc:492] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42: Bootstrap starting.
I20260809 07:55:48.831703 16362 tablet_service.cc:1467] Processing CreateTablet for tablet 364d7d86555a4878a78a53c31b2db9e8 (DEFAULT_TABLE table=test-table [id=7716094a3de8448e89e315f277429cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:55:48.832962 16362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 364d7d86555a4878a78a53c31b2db9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:55:48.833204 16958 tablet_bootstrap.cc:654] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42: Neither blocks nor log segments found. Creating new log.
I20260809 07:55:48.834573 16958 log.cc:824] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:48.836450 16953 tablet_bootstrap.cc:492] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1: No bootstrap required, opened a new log
I20260809 07:55:48.836582 16958 tablet_bootstrap.cc:492] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42: No bootstrap required, opened a new log
I20260809 07:55:48.836761 16953 ts_tablet_manager.cc:1397] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260809 07:55:48.836759 16958 ts_tablet_manager.cc:1397] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260809 07:55:48.840732 16953 raft_consensus.cc:348] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d7563e79142d3a951e19acd78134c" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 33497 } } peers { permanent_uuid: "2c8d0183d9f14b9b9b17d135cbd88e42" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34949 } } peers { permanent_uuid: "29f1d457bdef499ca9946e83387f5ed1" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 37343 } }
I20260809 07:55:48.840732 16958 raft_consensus.cc:348] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d7563e79142d3a951e19acd78134c" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 33497 } } peers { permanent_uuid: "2c8d0183d9f14b9b9b17d135cbd88e42" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34949 } } peers { permanent_uuid: "29f1d457bdef499ca9946e83387f5ed1" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 37343 } }
I20260809 07:55:48.840999 16958 raft_consensus.cc:374] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:55:48.841054 16958 raft_consensus.cc:729] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c8d0183d9f14b9b9b17d135cbd88e42, State: Initialized, Role: FOLLOWER
I20260809 07:55:48.841185 16953 raft_consensus.cc:374] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:55:48.841300 16953 raft_consensus.cc:729] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f1d457bdef499ca9946e83387f5ed1, State: Initialized, Role: FOLLOWER
I20260809 07:55:48.841580 16958 consensus_queue.cc:260] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d7563e79142d3a951e19acd78134c" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 33497 } } peers { permanent_uuid: "2c8d0183d9f14b9b9b17d135cbd88e42" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34949 } } peers { permanent_uuid: "29f1d457bdef499ca9946e83387f5ed1" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 37343 } }
I20260809 07:55:48.842597 16961 tablet_bootstrap.cc:492] T 364d7d86555a4878a78a53c31b2db9e8 P b69d7563e79142d3a951e19acd78134c: Bootstrap starting.
I20260809 07:55:48.842695 16958 ts_tablet_manager.cc:1428] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260809 07:55:48.843065 16681 heartbeater.cc:503] Master 127.28.112.62:37947 was elected leader, sending a full tablet report...
I20260809 07:55:48.843091 16953 consensus_queue.cc:260] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d7563e79142d3a951e19acd78134c" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 33497 } } peers { permanent_uuid: "2c8d0183d9f14b9b9b17d135cbd88e42" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34949 } } peers { permanent_uuid: "29f1d457bdef499ca9946e83387f5ed1" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 37343 } }
I20260809 07:55:48.844730 16953 ts_tablet_manager.cc:1428] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260809 07:55:48.845070 16961 tablet_bootstrap.cc:654] T 364d7d86555a4878a78a53c31b2db9e8 P b69d7563e79142d3a951e19acd78134c: Neither blocks nor log segments found. Creating new log.
I20260809 07:55:48.845585 16898 heartbeater.cc:503] Master 127.28.112.62:37947 was elected leader, sending a full tablet report...
I20260809 07:55:48.846060 16961 log.cc:824] T 364d7d86555a4878a78a53c31b2db9e8 P b69d7563e79142d3a951e19acd78134c: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:48.849747 16961 tablet_bootstrap.cc:492] T 364d7d86555a4878a78a53c31b2db9e8 P b69d7563e79142d3a951e19acd78134c: No bootstrap required, opened a new log
I20260809 07:55:48.849990 16961 ts_tablet_manager.cc:1397] T 364d7d86555a4878a78a53c31b2db9e8 P b69d7563e79142d3a951e19acd78134c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 07:55:48.853528 16961 raft_consensus.cc:348] T 364d7d86555a4878a78a53c31b2db9e8 P b69d7563e79142d3a951e19acd78134c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d7563e79142d3a951e19acd78134c" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 33497 } } peers { permanent_uuid: "2c8d0183d9f14b9b9b17d135cbd88e42" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34949 } } peers { permanent_uuid: "29f1d457bdef499ca9946e83387f5ed1" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 37343 } }
I20260809 07:55:48.853895 16961 raft_consensus.cc:374] T 364d7d86555a4878a78a53c31b2db9e8 P b69d7563e79142d3a951e19acd78134c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:55:48.853992 16961 raft_consensus.cc:729] T 364d7d86555a4878a78a53c31b2db9e8 P b69d7563e79142d3a951e19acd78134c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69d7563e79142d3a951e19acd78134c, State: Initialized, Role: FOLLOWER
I20260809 07:55:48.854465 16961 consensus_queue.cc:260] T 364d7d86555a4878a78a53c31b2db9e8 P b69d7563e79142d3a951e19acd78134c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d7563e79142d3a951e19acd78134c" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 33497 } } peers { permanent_uuid: "2c8d0183d9f14b9b9b17d135cbd88e42" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34949 } } peers { permanent_uuid: "29f1d457bdef499ca9946e83387f5ed1" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 37343 } }
I20260809 07:55:48.855536 16961 ts_tablet_manager.cc:1428] T 364d7d86555a4878a78a53c31b2db9e8 P b69d7563e79142d3a951e19acd78134c: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260809 07:55:48.855870 16465 heartbeater.cc:503] Master 127.28.112.62:37947 was elected leader, sending a full tablet report...
I20260809 07:55:48.861516 16964 raft_consensus.cc:482] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:55:48.861685 16964 raft_consensus.cc:504] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d7563e79142d3a951e19acd78134c" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 33497 } } peers { permanent_uuid: "2c8d0183d9f14b9b9b17d135cbd88e42" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34949 } } peers { permanent_uuid: "29f1d457bdef499ca9946e83387f5ed1" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 37343 } }
I20260809 07:55:48.862843 16964 leader_election.cc:288] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b69d7563e79142d3a951e19acd78134c (127.28.112.1:33497), 2c8d0183d9f14b9b9b17d135cbd88e42 (127.28.112.2:34949)
I20260809 07:55:48.872203 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:48.863167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:45871 (local address 127.28.112.1:33497)
0809 07:55:48.863697 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:48.863700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:48.867646 (+ 3946us) server_negotiation.cc:407] Connection header received
0809 07:55:48.867703 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:48.867706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:48.867752 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:48.867822 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:55:48.868473 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.869195 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.870933 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.871519 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.871630 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:48.871684 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:48.871995 (+ 311us) server_negotiation.cc:299] Negotiation successful
0809 07:55:48.872076 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467}
I20260809 07:55:48.872777 16971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:48.863071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:33497 (local address 127.28.112.3:45871)
0809 07:55:48.866674 (+ 3603us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:48.866689 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:48.866797 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:48.868200 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:48.868203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:48.868223 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:55:48.868337 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:48.868343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.869338 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.869341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:48.870799 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:48.870810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.871743 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.871747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:48.871840 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:48.871864 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 07:55:48.872530 (+ 666us) client_negotiation.cc:240] Negotiation successful
0809 07:55:48.872610 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3521,"mutex_wait_us":3,"thread_start_us":123,"threads_started":1}
I20260809 07:55:48.873252 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:48.863453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:43859 (local address 127.28.112.2:34949)
0809 07:55:48.863578 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:48.863582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:48.867073 (+ 3491us) server_negotiation.cc:407] Connection header received
0809 07:55:48.867252 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:48.867256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:48.867305 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:48.867351 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:55:48.868375 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.869194 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.872033 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.872597 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.872690 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:48.872737 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:48.873072 (+ 335us) server_negotiation.cc:299] Negotiation successful
0809 07:55:48.873158 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 07:55:48.873631 16380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364d7d86555a4878a78a53c31b2db9e8" candidate_uuid: "29f1d457bdef499ca9946e83387f5ed1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b69d7563e79142d3a951e19acd78134c" is_pre_election: true
I20260809 07:55:48.873777 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:48.863093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:34949 (local address 127.28.112.3:43859)
0809 07:55:48.866992 (+ 3899us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:48.867003 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:48.867140 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:48.867505 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:48.867508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:48.867526 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:55:48.868236 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:48.868245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.869338 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.869341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:48.871906 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:48.871914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.872832 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.872835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:48.872934 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:48.872955 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 07:55:48.873450 (+ 495us) client_negotiation.cc:240] Negotiation successful
0809 07:55:48.873641 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3838,"mutex_wait_us":1422,"thread_start_us":101,"threads_started":1}
I20260809 07:55:48.874271 16380 raft_consensus.cc:2454] T 364d7d86555a4878a78a53c31b2db9e8 P b69d7563e79142d3a951e19acd78134c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f1d457bdef499ca9946e83387f5ed1 in term 0.
I20260809 07:55:48.874447 16602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364d7d86555a4878a78a53c31b2db9e8" candidate_uuid: "29f1d457bdef499ca9946e83387f5ed1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c8d0183d9f14b9b9b17d135cbd88e42" is_pre_election: true
I20260809 07:55:48.874759 16718 leader_election.cc:302] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f1d457bdef499ca9946e83387f5ed1, b69d7563e79142d3a951e19acd78134c; no voters:
I20260809 07:55:48.874780 16602 raft_consensus.cc:2454] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f1d457bdef499ca9946e83387f5ed1 in term 0.
I20260809 07:55:48.875016 16964 raft_consensus.cc:2781] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:55:48.875097 16964 raft_consensus.cc:482] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:55:48.875145 16964 raft_consensus.cc:3037] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:55:48.876252 16964 raft_consensus.cc:504] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d7563e79142d3a951e19acd78134c" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 33497 } } peers { permanent_uuid: "2c8d0183d9f14b9b9b17d135cbd88e42" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34949 } } peers { permanent_uuid: "29f1d457bdef499ca9946e83387f5ed1" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 37343 } }
I20260809 07:55:48.876607 16964 leader_election.cc:288] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [CANDIDATE]: Term 1 election: Requested vote from peers b69d7563e79142d3a951e19acd78134c (127.28.112.1:33497), 2c8d0183d9f14b9b9b17d135cbd88e42 (127.28.112.2:34949)
I20260809 07:55:48.876904 16380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364d7d86555a4878a78a53c31b2db9e8" candidate_uuid: "29f1d457bdef499ca9946e83387f5ed1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b69d7563e79142d3a951e19acd78134c"
I20260809 07:55:48.876968 16602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364d7d86555a4878a78a53c31b2db9e8" candidate_uuid: "29f1d457bdef499ca9946e83387f5ed1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c8d0183d9f14b9b9b17d135cbd88e42"
I20260809 07:55:48.877029 16380 raft_consensus.cc:3037] T 364d7d86555a4878a78a53c31b2db9e8 P b69d7563e79142d3a951e19acd78134c [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:55:48.877066 16602 raft_consensus.cc:3037] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:55:48.878301 16380 raft_consensus.cc:2454] T 364d7d86555a4878a78a53c31b2db9e8 P b69d7563e79142d3a951e19acd78134c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f1d457bdef499ca9946e83387f5ed1 in term 1.
I20260809 07:55:48.878301 16602 raft_consensus.cc:2454] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f1d457bdef499ca9946e83387f5ed1 in term 1.
I20260809 07:55:48.878628 16716 leader_election.cc:302] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f1d457bdef499ca9946e83387f5ed1, 2c8d0183d9f14b9b9b17d135cbd88e42; no voters:
I20260809 07:55:48.878806 16964 raft_consensus.cc:2781] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:55:48.879019 16964 raft_consensus.cc:686] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [term 1 LEADER]: Becoming Leader. State: Replica: 29f1d457bdef499ca9946e83387f5ed1, State: Running, Role: LEADER
I20260809 07:55:48.879544 16964 consensus_queue.cc:237] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d7563e79142d3a951e19acd78134c" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 33497 } } peers { permanent_uuid: "2c8d0183d9f14b9b9b17d135cbd88e42" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34949 } } peers { permanent_uuid: "29f1d457bdef499ca9946e83387f5ed1" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 37343 } }
I20260809 07:55:48.882957 16137 catalog_manager.cc:5495] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f1d457bdef499ca9946e83387f5ed1 (127.28.112.3). New cstate: current_term: 1 leader_uuid: "29f1d457bdef499ca9946e83387f5ed1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d7563e79142d3a951e19acd78134c" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 33497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c8d0183d9f14b9b9b17d135cbd88e42" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f1d457bdef499ca9946e83387f5ed1" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 37343 } health_report { overall_health: HEALTHY } } }
I20260809 07:55:48.902835 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:48.897797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55312 (local address 127.28.112.3:37343)
0809 07:55:48.897911 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:48.897916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:48.897980 (+ 64us) server_negotiation.cc:407] Connection header received
0809 07:55:48.898211 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:48.898215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:48.898289 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:48.898399 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:55:48.898786 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.899460 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.900630 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:48.900923 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:48.901047 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:48.901847 (+ 800us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:55:48.902356 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:55:48.902449 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:48.902610 (+ 161us) server_negotiation.cc:299] Negotiation successful
0809 07:55:48.902676 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 07:55:48.910130 16602 raft_consensus.cc:1261] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42 [term 1 FOLLOWER]: Refusing update from remote peer 29f1d457bdef499ca9946e83387f5ed1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:55:48.910301 16380 raft_consensus.cc:1261] T 364d7d86555a4878a78a53c31b2db9e8 P b69d7563e79142d3a951e19acd78134c [term 1 FOLLOWER]: Refusing update from remote peer 29f1d457bdef499ca9946e83387f5ed1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:55:48.910960 16964 consensus_queue.cc:1035] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c8d0183d9f14b9b9b17d135cbd88e42" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 34949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:55:48.911229 16976 consensus_queue.cc:1035] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69d7563e79142d3a951e19acd78134c" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:55:48.921188 16979 mvcc.cc:205] Tried to move back new op lower bound from 7316529761924431872 to 7316529761813299200. Current Snapshot: MvccSnapshot[applied={T|T < 7316529761924431872}]
I20260809 07:55:48.924129 16982 mvcc.cc:205] Tried to move back new op lower bound from 7316529761924431872 to 7316529761813299200. Current Snapshot: MvccSnapshot[applied={T|T < 7316529761924431872}]
I20260809 07:55:48.938742 16137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:55:48.938935 16137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:55:48.941633 16137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c01864245f9b47c3b0bf1ae9f9143b71: Sending DeleteTablet for 3 replicas of tablet 364d7d86555a4878a78a53c31b2db9e8
I20260809 07:55:48.942894 16362 tablet_service.cc:1514] Processing DeleteTablet for tablet 364d7d86555a4878a78a53c31b2db9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:55:48 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:44322
I20260809 07:55:48.943454 16574 tablet_service.cc:1514] Processing DeleteTablet for tablet 364d7d86555a4878a78a53c31b2db9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:55:48 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:52492
I20260809 07:55:48.944303 16785 tablet_service.cc:1514] Processing DeleteTablet for tablet 364d7d86555a4878a78a53c31b2db9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:55:48 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:55310
I20260809 07:55:48.945454 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 16195
I20260809 07:55:48.952874 16995 mvcc.cc:205] Tried to move back new op lower bound from 7316529761924431872 to 7316529761813299200. Current Snapshot: MvccSnapshot[applied={T|T < 7316529761924431872}]
I20260809 07:55:48.967996 17000 tablet_replica.cc:330] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1: stopping tablet replica
I20260809 07:55:48.968297 17000 raft_consensus.cc:2229] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:55:48.968631 17000 raft_consensus.cc:2258] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:55:48.968943 16999 tablet_replica.cc:330] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42: stopping tablet replica
I20260809 07:55:48.969218 16999 raft_consensus.cc:2229] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:55:48.969519 16999 raft_consensus.cc:2258] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:55:48.970471 17000 ts_tablet_manager.cc:1905] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:55:48.973340 17000 ts_tablet_manager.cc:1918] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:55:48.974609 16999 ts_tablet_manager.cc:1905] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:55:48.977252 16999 ts_tablet_manager.cc:1918] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:55:48.977425 17000 log.cc:1196] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/364d7d86555a4878a78a53c31b2db9e8
I20260809 07:55:48.977779 17000 ts_tablet_manager.cc:1939] T 364d7d86555a4878a78a53c31b2db9e8 P 29f1d457bdef499ca9946e83387f5ed1: Deleting consensus metadata
I20260809 07:55:48.978847 16999 log.cc:1196] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/364d7d86555a4878a78a53c31b2db9e8
W20260809 07:55:48.978884 16113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.112.1:33497 (error 108)
I20260809 07:55:48.979296 16999 ts_tablet_manager.cc:1939] T 364d7d86555a4878a78a53c31b2db9e8 P 2c8d0183d9f14b9b9b17d135cbd88e42: Deleting consensus metadata
I20260809 07:55:48.979756 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 16470
I20260809 07:55:48.979980 16106 catalog_manager.cc:4841] TS 29f1d457bdef499ca9946e83387f5ed1 (127.28.112.3:37343): tablet 364d7d86555a4878a78a53c31b2db9e8 (table test-table [id=7716094a3de8448e89e315f277429cc9]) successfully deleted
I20260809 07:55:48.985874 16945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:48.985496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:33497 (local address 127.0.0.1:44324)
0809 07:55:48.985715 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:48.985786 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:33497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260809 07:55:48.986461 16113 catalog_manager.cc:4584] TS b69d7563e79142d3a951e19acd78134c (127.28.112.1:33497): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 364d7d86555a4878a78a53c31b2db9e8: Network error: Client connection negotiation failed: client connection to 127.28.112.1:33497: connect: Connection refused (error 111)
W20260809 07:55:48.993749 16945 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:55:48.993091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:34949 (local address 127.0.0.1:52508)
0809 07:55:48.993277 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:48.993293 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:48.993387 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:48.993631 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.2:34949: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":109}
I20260809 07:55:48.994405 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 16686
I20260809 07:55:49.012795 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 16078
I20260809 07:55:49.025187 29120 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 07:55:49.369362 29120 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 07:55:49.369473 29120 mini_ranger.cc:66] Stopping Ranger...
I20260809 07:55:49.713776 29120 mini_ranger.cc:68] Stopped Ranger
2026-08-09 07:55:49.713 UTC [14602] LOG: received smart shutdown request
2026-08-09 07:55:49.718 UTC [14602] LOG: background worker "logical replication launcher" (PID 14620) exited with exit code 1
2026-08-09 07:55:49.718 UTC [14610] LOG: shutting down
2026-08-09 07:55:49.858 UTC [14602] LOG: database system is shut down
2026-08-09T07:55:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47934 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:55:50 dist-test-slave-vpvm krb5kdc[17023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:55:50 dist-test-slave-vpvm krb5kdc[17023](info): set up 2 sockets
Aug 09 07:55:50 dist-test-slave-vpvm krb5kdc[17023](info): commencing operation
krb5kdc: starting...
W20260809 07:55:51.098253 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:55:51 dist-test-slave-vpvm krb5kdc[17023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:55:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:55:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:55:51.213920 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:45507
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:33601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:45507,127.28.112.61:39571,127.28.112.60:37387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:55:51.322593 17042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:51.322860 17042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:51.322917 17042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:51.326859 17042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:55:51.326915 17042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:51.326941 17042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:51.326963 17042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:55:51.326992 17042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:55:51.332149 17042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:33601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:45507,127.28.112.61:39571,127.28.112.60:37387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:45507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:51.333158 17042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:51.334214 17042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:51.342200 17052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:51.342250 17057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:51.342244 17053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:51.342579 17042 server_base.cc:1034] running on GCE node
I20260809 07:55:51.343122 17042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:51.344197 17042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:51.345366 17042 hybrid_clock.cc:648] HybridClock initialized: now 1786262151345341 us; error 31 us; skew 500 ppm
Aug 09 07:55:51 dist-test-slave-vpvm krb5kdc[17023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:51.348173 17042 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:55:51.349051 17042 webserver.cc:460] Webserver started at http://127.28.112.62:44127/ using document root <none> and password file <none>
I20260809 07:55:51.349574 17042 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:51.349634 17042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:51.349836 17042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:55:51.351944 17042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "14942c97345640798205d02047c2eae0"
format_stamp: "Formatted at 2026-08-09 07:55:51 on dist-test-slave-vpvm"
server_key: "0ac680733927c9e18e50b5a0845f0c46"
server_key_iv: "0b5f0f0db85a20b257dff4bba55c2df6"
server_key_version: "encryptionkey@0"
I20260809 07:55:51.352496 17042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "14942c97345640798205d02047c2eae0"
format_stamp: "Formatted at 2026-08-09 07:55:51 on dist-test-slave-vpvm"
server_key: "0ac680733927c9e18e50b5a0845f0c46"
server_key_iv: "0b5f0f0db85a20b257dff4bba55c2df6"
server_key_version: "encryptionkey@0"
I20260809 07:55:51.356073 17042 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 07:55:51.358338 17066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:51.359513 17042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:55:51.359618 17042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "14942c97345640798205d02047c2eae0"
format_stamp: "Formatted at 2026-08-09 07:55:51 on dist-test-slave-vpvm"
server_key: "0ac680733927c9e18e50b5a0845f0c46"
server_key_iv: "0b5f0f0db85a20b257dff4bba55c2df6"
server_key_version: "encryptionkey@0"
I20260809 07:55:51.359711 17042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:51.376825 17042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:51.379927 17042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:51.380100 17042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:51.386595 17042 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:45507
I20260809 07:55:51.386607 17142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:45507 every 8 connection(s)
I20260809 07:55:51.387554 17042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:55:51.389653 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 17042
I20260809 07:55:51.389766 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:55:51.390065 29120 external_mini_cluster.cc:1428] Setting key 20ecaa59130de3cba47a9f8aae75266c
I20260809 07:55:51.390322 17145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:55:51.394944 17145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.112.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.61@KRBTEST.COM" created.
Aug 09 07:55:51 dist-test-slave-vpvm krb5kdc[17023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:55:51.409757 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.397492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:45507 (local address 127.0.0.1:41582)
0809 07:55:51.398000 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.398049 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:51.398827 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:51.399571 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:51.399575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:51.399869 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:51.400125 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:51.400139 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.401263 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.401266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:51.402122 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:51.402135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.402872 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.402875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:51.402970 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.403492 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:51.403508 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:51.405078 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:51.406963 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:51.406971 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:51.406981 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:51.407201 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:51.407488 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:51.407491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:51.407493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:51.407602 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:51.408157 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:51.408163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:51.408291 (+ 128us) client_negotiation.cc:769] Sending connection context
0809 07:55:51.408435 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 07:55:51.408807 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":89,"thread_start_us":123,"threads_started":1}
I20260809 07:55:51.409757 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.397492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41582 (local address 127.28.112.62:45507)
0809 07:55:51.398036 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:51.398055 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:51.398156 (+ 101us) server_negotiation.cc:407] Connection header received
0809 07:55:51.399131 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:51.399137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:51.399366 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:51.399584 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:51.400281 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.401135 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.402231 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.402698 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.402854 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.405228 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:51.405234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:51.405245 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:51.405267 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:51.406860 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.407301 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.407305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.407307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.407404 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.407683 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.407687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.407689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.408035 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:51.408168 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:51.408510 (+ 342us) server_negotiation.cc:299] Negotiation successful
0809 07:55:51.408743 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":401,"spinlock_wait_cycles":17152,"thread_start_us":155,"threads_started":1}
I20260809 07:55:51.412133 17145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } has no permanent_uuid. Determining permanent_uuid...
I20260809 07:55:51.412802 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.412570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:39571 (local address 127.0.0.1:32974)
0809 07:55:51.412676 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.412742 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:39571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 07:55:51.412976 17071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.112.61:39571: connect: Connection refused (error 111)
I20260809 07:55:51.413764 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.413551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:39571 (local address 127.0.0.1:32990)
0809 07:55:51.413666 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.413710 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:39571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260809 07:55:51.413908 17145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.61:39571: Network error: Client connection negotiation failed: client connection to 127.28.112.61:39571: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.61.keytab.
Entry for principal kudu/127.28.112.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.61.keytab.
I20260809 07:55:51.417719 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.61:39571
--webserver_interface=127.28.112.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.61.keytab
--principal=kudu/127.28.112.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:33601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:45507,127.28.112.61:39571,127.28.112.60:37387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 07:55:51.463156 17145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } attempt: 1
I20260809 07:55:51.464064 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.463713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:39571 (local address 127.0.0.1:32994)
0809 07:55:51.463872 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.463947 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:39571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 07:55:51.464745 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.464525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:39571 (local address 127.0.0.1:32998)
0809 07:55:51.464620 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.464678 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:39571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260809 07:55:51.464872 17145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.61:39571: Network error: Client connection negotiation failed: client connection to 127.28.112.61:39571: connect: Connection refused (error 111)
W20260809 07:55:51.527386 17160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:51.527630 17160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:51.527668 17160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:51.531538 17160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:55:51.531594 17160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:51.531613 17160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:51.531628 17160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:55:51.531642 17160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:55:51.533031 17145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } attempt: 2
I20260809 07:55:51.533932 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.533602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:39571 (local address 127.0.0.1:33006)
0809 07:55:51.533769 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.533839 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:39571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 07:55:51.534669 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.534452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:39571 (local address 127.0.0.1:33014)
0809 07:55:51.534546 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.534595 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:39571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260809 07:55:51.534843 17145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.61:39571: Network error: Client connection negotiation failed: client connection to 127.28.112.61:39571: connect: Connection refused (error 111)
I20260809 07:55:51.537185 17160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:33601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:45507,127.28.112.61:39571,127.28.112.60:37387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.61.keytab
--principal=kudu/127.28.112.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.61:39571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.112.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:51.538302 17160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:51.539322 17160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:51.547652 17168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:51.547643 17167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:51.547653 17160 server_base.cc:1034] running on GCE node
W20260809 07:55:51.547668 17172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:51.548408 17160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:51.549491 17160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:51.550652 17160 hybrid_clock.cc:648] HybridClock initialized: now 1786262151550636 us; error 30 us; skew 500 ppm
Aug 09 07:55:51 dist-test-slave-vpvm krb5kdc[17023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:51.553323 17160 init.cc:377] Logged in from keytab as kudu/127.28.112.61@KRBTEST.COM (short username kudu)
I20260809 07:55:51.554244 17160 webserver.cc:460] Webserver started at http://127.28.112.61:41381/ using document root <none> and password file <none>
I20260809 07:55:51.554764 17160 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:51.554829 17160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:51.555047 17160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:55:51.557040 17160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-1/data/instance:
uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0"
format_stamp: "Formatted at 2026-08-09 07:55:51 on dist-test-slave-vpvm"
server_key: "4abc8ebb28b7bb7a2a946748738a705a"
server_key_iv: "c24ff355f11b823990a1c64e2b2fdcbf"
server_key_version: "encryptionkey@0"
I20260809 07:55:51.557554 17160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-1/wal/instance:
uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0"
format_stamp: "Formatted at 2026-08-09 07:55:51 on dist-test-slave-vpvm"
server_key: "4abc8ebb28b7bb7a2a946748738a705a"
server_key_iv: "c24ff355f11b823990a1c64e2b2fdcbf"
server_key_version: "encryptionkey@0"
I20260809 07:55:51.561041 17160 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:55:51.563438 17185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:51.564577 17160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:55:51.564685 17160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-1/wal
uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0"
format_stamp: "Formatted at 2026-08-09 07:55:51 on dist-test-slave-vpvm"
server_key: "4abc8ebb28b7bb7a2a946748738a705a"
server_key_iv: "c24ff355f11b823990a1c64e2b2fdcbf"
server_key_version: "encryptionkey@0"
I20260809 07:55:51.564770 17160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:51.579798 17160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:51.582195 17160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:51.582348 17160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:51.588773 17160 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.61:39571
I20260809 07:55:51.588805 17262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.61:39571 every 8 connection(s)
I20260809 07:55:51.589629 17160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-1/data/info.pb
I20260809 07:55:51.592260 17263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:55:51.593573 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 17160
I20260809 07:55:51.593703 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-1/wal/instance
I20260809 07:55:51.593974 29120 external_mini_cluster.cc:1428] Setting key 6096a491029d915000be4d6259a05a70
I20260809 07:55:51.596050 17263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.112.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.60@KRBTEST.COM" created.
Aug 09 07:55:51 dist-test-slave-vpvm krb5kdc[17023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.61@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:55:51.610014 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.598235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41598 (local address 127.28.112.62:45507)
0809 07:55:51.598380 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:51.598386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:51.599307 (+ 921us) server_negotiation.cc:407] Connection header received
0809 07:55:51.600454 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:51.600457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:51.600506 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:51.600566 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:51.602086 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.602632 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.603966 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.604211 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.604291 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.606666 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:51.606669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:51.606672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:51.606703 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:51.608039 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.608524 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.608527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.608529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.608605 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.608922 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.608926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.608928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.609102 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:51.609193 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:51.609698 (+ 505us) server_negotiation.cc:299] Negotiation successful
0809 07:55:51.609826 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 07:55:51.610751 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.598586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:45507 (local address 127.0.0.1:41598)
0809 07:55:51.599125 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.599193 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:51.600216 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:51.600765 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:51.600775 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:51.601255 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:51.601913 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:51.601933 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.602756 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.602760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:51.603816 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:51.603827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.604362 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.604365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:51.604515 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.605016 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:51.605038 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:51.606488 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:51.608162 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:51.608168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:51.608183 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:51.608427 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:51.608697 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:51.608700 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:51.608702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:51.608833 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:51.609205 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:51.609211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:51.609452 (+ 241us) client_negotiation.cc:769] Sending connection context
0809 07:55:51.609613 (+ 161us) client_negotiation.cc:240] Negotiation successful
0809 07:55:51.609917 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":124,"threads_started":1}
I20260809 07:55:51.612197 17263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 07:55:51 dist-test-slave-vpvm krb5kdc[17023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.61@KRBTEST.COM for kudu/127.28.112.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.60.keytab.
Entry for principal kudu/127.28.112.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.60.keytab.
I20260809 07:55:51.620628 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.612689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:39571 (local address 127.0.0.1:33022)
0809 07:55:51.613007 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.613021 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:51.613102 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:51.613443 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:51.613446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:51.613606 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:51.613678 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:51.613683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.614683 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.614686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:51.615205 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:51.615213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.615753 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.615756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:51.615888 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.616336 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:51.616348 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:51.617609 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:51.619251 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:51.619254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:51.619256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:51.619429 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:51.619680 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:51.619683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:51.619685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:51.619760 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:51.620179 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:51.620182 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:51.620267 (+ 85us) client_negotiation.cc:769] Sending connection context
0809 07:55:51.620355 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 07:55:51.620475 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":19}
I20260809 07:55:51.620654 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.60:37387
--webserver_interface=127.28.112.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.60.keytab
--principal=kudu/127.28.112.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:33601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:45507,127.28.112.61:39571,127.28.112.60:37387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 07:55:51.620711 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.612760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33022 (local address 127.28.112.61:39571)
0809 07:55:51.613009 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:51.613021 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:51.613071 (+ 50us) server_negotiation.cc:407] Connection header received
0809 07:55:51.613188 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:51.613194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:51.613337 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:51.613452 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:51.613775 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.614581 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.615306 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.615654 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.615745 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.617726 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:51.617734 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:51.617746 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:51.617766 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:51.619155 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.619523 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.619529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.619530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.619600 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.619834 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.619837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.619839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.620080 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:51.620197 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:51.620443 (+ 246us) server_negotiation.cc:299] Negotiation successful
0809 07:55:51.620562 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
I20260809 07:55:51.622635 17263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } has no permanent_uuid. Determining permanent_uuid...
I20260809 07:55:51.623284 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.623056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:37387 (local address 127.0.0.1:50700)
0809 07:55:51.623164 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.623230 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 07:55:51.623481 17187 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.112.60:37387: connect: Connection refused (error 111)
I20260809 07:55:51.624264 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.624041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:37387 (local address 127.0.0.1:50714)
0809 07:55:51.624152 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.624203 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 07:55:51.624405 17263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.60:37387: Network error: Client connection negotiation failed: client connection to 127.28.112.60:37387: connect: Connection refused (error 111)
I20260809 07:55:51.626039 17145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } attempt: 3
Aug 09 07:55:51 dist-test-slave-vpvm krb5kdc[17023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for kudu/127.28.112.61@KRBTEST.COM
I20260809 07:55:51.634399 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.626531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33034 (local address 127.28.112.61:39571)
0809 07:55:51.626781 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:51.626784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:51.626793 (+ 9us) server_negotiation.cc:407] Connection header received
0809 07:55:51.626818 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:51.626820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:51.626851 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:51.626903 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:51.627397 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.627869 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.629431 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.629651 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.629717 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.631701 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:51.631705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:51.631708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:51.631731 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:51.632931 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.633323 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.633327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.633328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.633396 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.633650 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.633654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.633656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.633808 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:51.633891 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:51.634171 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 07:55:51.634266 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260809 07:55:51.634404 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.626470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:39571 (local address 127.0.0.1:33034)
0809 07:55:51.626572 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.626586 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:51.626669 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:51.626964 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:51.626968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:51.627146 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:51.627277 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:51.627286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.628003 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.628008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:51.629318 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:51.629331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.629751 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.629755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:51.629862 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.630293 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:51.630308 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:51.631573 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:51.633034 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:51.633037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:51.633039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:51.633235 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:51.633478 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:51.633482 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:51.633484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:51.633563 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:51.633900 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:51.633903 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:51.634019 (+ 116us) client_negotiation.cc:769] Sending connection context
0809 07:55:51.634110 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 07:55:51.634252 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260809 07:55:51.635397 17145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } has no permanent_uuid. Determining permanent_uuid...
I20260809 07:55:51.636016 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.635790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:37387 (local address 127.0.0.1:50716)
0809 07:55:51.635898 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.635962 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260809 07:55:51.636667 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.636445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:37387 (local address 127.0.0.1:50730)
0809 07:55:51.636576 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.636617 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260809 07:55:51.636798 17145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.60:37387: Network error: Client connection negotiation failed: client connection to 127.28.112.60:37387: connect: Connection refused (error 111)
I20260809 07:55:51.667963 17145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } attempt: 1
I20260809 07:55:51.668646 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.668381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:37387 (local address 127.0.0.1:50746)
0809 07:55:51.668525 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.668581 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 07:55:51.669346 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.669120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:37387 (local address 127.0.0.1:50752)
0809 07:55:51.669231 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.669284 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 07:55:51.669476 17145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.60:37387: Network error: Client connection negotiation failed: client connection to 127.28.112.60:37387: connect: Connection refused (error 111)
I20260809 07:55:51.673651 17263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } attempt: 1
I20260809 07:55:51.674321 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.674073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:37387 (local address 127.0.0.1:50756)
0809 07:55:51.674193 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.674251 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260809 07:55:51.674939 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.674733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:37387 (local address 127.0.0.1:50770)
0809 07:55:51.674827 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.674875 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260809 07:55:51.675047 17263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.60:37387: Network error: Client connection negotiation failed: client connection to 127.28.112.60:37387: connect: Connection refused (error 111)
W20260809 07:55:51.730818 17271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:51.731051 17271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:51.731087 17271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:51.734915 17271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:55:51.734963 17271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:51.734982 17271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:51.734997 17271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:55:51.735009 17271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:55:51.740052 17271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:33601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:45507,127.28.112.61:39571,127.28.112.60:37387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.60.keytab
--principal=kudu/127.28.112.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.60:37387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.112.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:51.740980 17271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:51.741891 17271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 07:55:51.743196 17263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } attempt: 2
I20260809 07:55:51.744057 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.743723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:37387 (local address 127.0.0.1:50778)
0809 07:55:51.743884 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.743966 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 07:55:51.744627 17145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } attempt: 2
I20260809 07:55:51.744802 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.744547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:37387 (local address 127.0.0.1:50788)
0809 07:55:51.744678 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.744732 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260809 07:55:51.744930 17263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.60:37387: Network error: Client connection negotiation failed: client connection to 127.28.112.60:37387: connect: Connection refused (error 111)
I20260809 07:55:51.745318 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.745037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:37387 (local address 127.0.0.1:50796)
0809 07:55:51.745174 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.745243 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260809 07:55:51.746004 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.745781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:37387 (local address 127.0.0.1:50806)
0809 07:55:51.745868 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.745913 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260809 07:55:51.746163 17145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.60:37387: Network error: Client connection negotiation failed: client connection to 127.28.112.60:37387: connect: Connection refused (error 111)
W20260809 07:55:51.750401 17278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:51.750398 17282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:51.750398 17280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:51.750823 17271 server_base.cc:1034] running on GCE node
I20260809 07:55:51.751379 17271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:51.752367 17271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:51.753530 17271 hybrid_clock.cc:648] HybridClock initialized: now 1786262151753512 us; error 28 us; skew 500 ppm
Aug 09 07:55:51 dist-test-slave-vpvm krb5kdc[17023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:51.756095 17271 init.cc:377] Logged in from keytab as kudu/127.28.112.60@KRBTEST.COM (short username kudu)
I20260809 07:55:51.756906 17271 webserver.cc:460] Webserver started at http://127.28.112.60:45975/ using document root <none> and password file <none>
I20260809 07:55:51.757412 17271 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:51.757472 17271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:51.757660 17271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:55:51.759594 17271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-2/data/instance:
uuid: "6e72a2f5b0674ae58763e956bbeff370"
format_stamp: "Formatted at 2026-08-09 07:55:51 on dist-test-slave-vpvm"
server_key: "2dd1cc56f26ac16e2fe11c51e392e0e7"
server_key_iv: "e91a0d7e3a2856474a0336bea5322cfb"
server_key_version: "encryptionkey@0"
I20260809 07:55:51.760105 17271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-2/wal/instance:
uuid: "6e72a2f5b0674ae58763e956bbeff370"
format_stamp: "Formatted at 2026-08-09 07:55:51 on dist-test-slave-vpvm"
server_key: "2dd1cc56f26ac16e2fe11c51e392e0e7"
server_key_iv: "e91a0d7e3a2856474a0336bea5322cfb"
server_key_version: "encryptionkey@0"
I20260809 07:55:51.763576 17271 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260809 07:55:51.765822 17293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:51.766974 17271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 07:55:51.767073 17271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-2/wal
uuid: "6e72a2f5b0674ae58763e956bbeff370"
format_stamp: "Formatted at 2026-08-09 07:55:51 on dist-test-slave-vpvm"
server_key: "2dd1cc56f26ac16e2fe11c51e392e0e7"
server_key_iv: "e91a0d7e3a2856474a0336bea5322cfb"
server_key_version: "encryptionkey@0"
I20260809 07:55:51.767168 17271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:51.779228 17271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:51.781807 17271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:51.782001 17271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:51.788543 17271 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.60:37387
I20260809 07:55:51.788558 17375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.60:37387 every 8 connection(s)
I20260809 07:55:51.789415 17271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-2/data/info.pb
I20260809 07:55:51.792145 17376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:55:51.795804 17376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } has no permanent_uuid. Determining permanent_uuid...
I20260809 07:55:51.797004 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 17271
I20260809 07:55:51.797137 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786261991120089-29120-0/minicluster-data/master-2/wal/instance
I20260809 07:55:51.797451 29120 external_mini_cluster.cc:1428] Setting key 07fbe67cd840eb4405cb367bc9b8cacd
Aug 09 07:55:51 dist-test-slave-vpvm krb5kdc[17023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.60@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
Aug 09 07:55:51 dist-test-slave-vpvm krb5kdc[17023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.60@KRBTEST.COM
I20260809 07:55:51.808508 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.797540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.28.112.62:45507)
0809 07:55:51.797667 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:51.797671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:51.798697 (+ 1026us) server_negotiation.cc:407] Connection header received
0809 07:55:51.799433 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:51.799437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:51.799491 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:51.799562 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:51.800947 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.801474 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.802652 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.802884 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.802985 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.805187 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:51.805190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:51.805192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:51.805217 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:51.806575 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.807071 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.807075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.807077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.807159 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.807468 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.807472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.807474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.807675 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:51.807763 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:51.808230 (+ 467us) server_negotiation.cc:299] Negotiation successful
0809 07:55:51.808348 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 07:55:51.809278 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.797991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:45507 (local address 127.0.0.1:41600)
0809 07:55:51.798555 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.798606 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:51.799251 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:51.800354 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:51.800359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:51.800662 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:51.800809 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:51.800818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.801603 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.801606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:51.802535 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:51.802546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.803023 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.803026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:51.803103 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.803557 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:51.803581 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:51.805044 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:51.806679 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:51.806688 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:51.806697 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:51.806980 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:51.807243 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:51.807247 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:51.807249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:51.807379 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:51.807778 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:51.807785 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:51.808011 (+ 226us) client_negotiation.cc:769] Sending connection context
0809 07:55:51.808177 (+ 166us) client_negotiation.cc:240] Negotiation successful
0809 07:55:51.808457 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":3,"thread_start_us":136,"threads_started":1}
I20260809 07:55:51.809283 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.798098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50816 (local address 127.28.112.60:37387)
0809 07:55:51.798554 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:51.798571 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:51.798608 (+ 37us) server_negotiation.cc:407] Connection header received
0809 07:55:51.799243 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:51.799260 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:51.799451 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:51.799588 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:51.800335 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.801257 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.802171 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.802750 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.802856 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.805561 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:51.805568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:51.805580 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:51.805606 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:51.807108 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.807520 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.807526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.807527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.807597 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.807840 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.807843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.807845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.808070 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:51.808238 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:51.808454 (+ 216us) server_negotiation.cc:299] Negotiation successful
0809 07:55:51.808580 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":322,"spinlock_wait_cycles":42880,"thread_start_us":139,"threads_started":1}
I20260809 07:55:51.810425 17376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 07:55:51 dist-test-slave-vpvm krb5kdc[17023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.60@KRBTEST.COM for kudu/127.28.112.61@KRBTEST.COM
Aug 09 07:55:51 dist-test-slave-vpvm krb5kdc[17023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:55:51.818243 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.810827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:39571 (local address 127.0.0.1:33050)
0809 07:55:51.811046 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.811059 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:51.811136 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:51.811339 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:51.811342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:51.811492 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:51.811597 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:51.811605 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.812384 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.812387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:51.812918 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:51.812928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.813321 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.813324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:51.813422 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.813830 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:51.813841 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:51.815237 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:51.816818 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:51.816821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:51.816823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:51.817048 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:51.817328 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:51.817331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:51.817334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:51.817424 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:51.817770 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:51.817773 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:51.817862 (+ 89us) client_negotiation.cc:769] Sending connection context
0809 07:55:51.817979 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 07:55:51.818101 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260809 07:55:51.818292 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.810937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33050 (local address 127.28.112.61:39571)
0809 07:55:51.811047 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:51.811051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:51.811100 (+ 49us) server_negotiation.cc:407] Connection header received
0809 07:55:51.811207 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:51.811211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:51.811253 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:51.811319 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:51.811728 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.812275 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.813026 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.813230 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.813302 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.815375 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:51.815379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:51.815381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:51.815409 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:51.816717 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.817145 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.817149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.817150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.817241 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.817518 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.817520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.817522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.817683 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:51.817755 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:51.818047 (+ 292us) server_negotiation.cc:299] Negotiation successful
0809 07:55:51.818162 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 07:55:51.818857 17376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } has no permanent_uuid. Determining permanent_uuid...
I20260809 07:55:51.819380 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.811470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41602 (local address 127.28.112.62:45507)
0809 07:55:51.811567 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:51.811571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:51.811583 (+ 12us) server_negotiation.cc:407] Connection header received
0809 07:55:51.811698 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:51.811701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:51.811739 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:51.811812 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:51.812310 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.812789 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.813657 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.813940 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.814050 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.816346 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:51.816349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:51.816351 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:51.816372 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:51.817781 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.818195 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.818198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.818200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.818273 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.818571 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.818575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.818576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.818791 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:51.818994 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:51.819132 (+ 138us) server_negotiation.cc:299] Negotiation successful
0809 07:55:51.819232 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Aug 09 07:55:51 dist-test-slave-vpvm krb5kdc[17023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.60@KRBTEST.COM for kudu/127.28.112.60@KRBTEST.COM
Aug 09 07:55:51 dist-test-slave-vpvm krb5kdc[17023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.61@KRBTEST.COM
I20260809 07:55:51.826367 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.819277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:37387 (local address 127.0.0.1:50818)
0809 07:55:51.819618 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.819630 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:51.819706 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:51.819899 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:51.819902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:51.820046 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:51.820130 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:51.820136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.821022 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.821025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:51.821498 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:51.821505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.821902 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.821906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:51.821999 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.822373 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:51.822383 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:51.823397 (+ 1014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:51.824951 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:51.824955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:51.824956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:51.825135 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:51.825394 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:51.825397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:51.825399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:51.825481 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:51.825858 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:51.825862 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:51.825988 (+ 126us) client_negotiation.cc:769] Sending connection context
0809 07:55:51.826091 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 07:55:51.826195 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279}
I20260809 07:55:51.826388 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.819358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.28.112.60:37387)
0809 07:55:51.819492 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:51.819496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:51.819696 (+ 200us) server_negotiation.cc:407] Connection header received
0809 07:55:51.819759 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:51.819761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:51.819802 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:51.819870 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:51.820405 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.820927 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.821594 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.821813 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.821893 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.823509 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:51.823512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:51.823515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:51.823535 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:51.824818 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.825225 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.825228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.825230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.825316 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.825569 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.825572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.825573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.825771 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:51.825846 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:51.826169 (+ 323us) server_negotiation.cc:299] Negotiation successful
0809 07:55:51.826272 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 07:55:51.828606 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.820599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33058 (local address 127.28.112.61:39571)
0809 07:55:51.820696 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:51.820700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:51.820713 (+ 13us) server_negotiation.cc:407] Connection header received
0809 07:55:51.820777 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:51.820779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:51.820814 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:51.820873 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:51.821397 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.821846 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.822683 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.822886 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.822968 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.825313 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:51.825315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:51.825317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:51.825337 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:51.827054 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.827597 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.827600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.827601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.827684 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.827907 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.827910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.827912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.828097 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:51.828299 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:51.828365 (+ 66us) server_negotiation.cc:299] Negotiation successful
0809 07:55:51.828471 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 07:55:51.829756 17376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370: Bootstrap starting.
I20260809 07:55:51.831700 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:55:51.832146 17376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370: Neither blocks nor log segments found. Creating new log.
I20260809 07:55:51.832823 17376 log.cc:824] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:51.835742 17376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370: No bootstrap required, opened a new log
I20260809 07:55:51.836128 17263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } attempt: 3
I20260809 07:55:51.841616 17376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } } peers { permanent_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } } peers { permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } }
I20260809 07:55:51.842267 17376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:55:51.842377 17376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e72a2f5b0674ae58763e956bbeff370, State: Initialized, Role: FOLLOWER
I20260809 07:55:51.842912 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.833899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41608 (local address 127.28.112.62:45507)
0809 07:55:51.834038 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:51.834042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:51.834054 (+ 12us) server_negotiation.cc:407] Connection header received
0809 07:55:51.834086 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:51.834089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:51.834129 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:51.834195 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:51.834957 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.835666 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.836657 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.836948 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.837024 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.838638 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:51.838641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:51.838644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:51.838668 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:51.840319 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.841544 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.841547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.841549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.841640 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.841944 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.841979 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.841981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.842180 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:51.842281 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:51.842650 (+ 369us) server_negotiation.cc:299] Negotiation successful
0809 07:55:51.842758 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 07:55:51.843183 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.833640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33064 (local address 127.28.112.61:39571)
0809 07:55:51.833731 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:51.833735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:51.834186 (+ 451us) server_negotiation.cc:407] Connection header received
0809 07:55:51.835091 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:51.835094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:51.835134 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:51.835179 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:51.836018 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.836678 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.837759 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.838110 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.838184 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.839709 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:51.839713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:51.839715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:51.839741 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:51.841330 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.841798 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.841801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.841803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.841892 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.842233 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.842236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.842238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.842439 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:51.842552 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:51.842907 (+ 355us) server_negotiation.cc:299] Negotiation successful
0809 07:55:51.843032 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 07:55:51.843418 17376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } } peers { permanent_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } } peers { permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } }
I20260809 07:55:51.845214 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.834052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50820 (local address 127.28.112.60:37387)
0809 07:55:51.834149 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:51.834153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:51.834470 (+ 317us) server_negotiation.cc:407] Connection header received
0809 07:55:51.834565 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:51.834568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:51.834604 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:51.834663 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:51.835363 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.836052 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.838046 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.838378 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.838450 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.841643 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:51.841647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:51.841649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:51.841672 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:51.843209 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.843658 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.843662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.843663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.843747 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.843991 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.843994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.843996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.844228 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:51.844338 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:51.844979 (+ 641us) server_negotiation.cc:299] Negotiation successful
0809 07:55:51.845087 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 07:55:51.845245 17376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370 [sys.catalog]: configured and running, proceeding with master startup.
Aug 09 07:55:51 dist-test-slave-vpvm krb5kdc[17023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.61@KRBTEST.COM for kudu/127.28.112.60@KRBTEST.COM
I20260809 07:55:51.845779 17145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } attempt: 3
I20260809 07:55:51.846517 17400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } } peers { permanent_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } } peers { permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } } }
I20260809 07:55:51.846725 17400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:55:51.849802 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.836437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:37387 (local address 127.0.0.1:50832)
0809 07:55:51.836643 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.836655 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:51.836716 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:51.839241 (+ 2525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:51.839244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:51.839397 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:51.839573 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:51.839587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.840707 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.840711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:51.841517 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:51.841529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.844181 (+ 2652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.844184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:51.844263 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.844628 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:51.844641 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:51.846288 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:51.848204 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:51.848207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:51.848209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:51.848457 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:51.848799 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:51.848802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:51.848803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:51.848891 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:51.849320 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:51.849323 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:51.849410 (+ 87us) client_negotiation.cc:769] Sending connection context
0809 07:55:51.849538 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 07:55:51.849661 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260809 07:55:51.850551 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.836842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50832 (local address 127.28.112.60:37387)
0809 07:55:51.839000 (+ 2158us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:51.839009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:51.839021 (+ 12us) server_negotiation.cc:407] Connection header received
0809 07:55:51.839052 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:51.839055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:51.839104 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:51.839708 (+ 604us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:51.839838 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.840579 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.841975 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.844059 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.844451 (+ 392us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.846433 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:51.846437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:51.846440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:51.846464 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:51.848107 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.848572 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.848576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.848577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.848663 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.848986 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.848990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.848991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.849185 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:51.849275 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:51.850279 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0809 07:55:51.850386 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2093,"thread_start_us":101,"threads_started":1}
W20260809 07:55:51.851648 17416 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:55:51.851768 17416 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 09 07:55:51 dist-test-slave-vpvm krb5kdc[17023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for kudu/127.28.112.60@KRBTEST.COM
I20260809 07:55:51.849049 17271 master_runner.cc:186] Error getting master registration for 127.28.112.62:45507: OK, instance_id { permanent_uuid: "14942c97345640798205d02047c2eae0" instance_seqno: 1786262151384804 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 07:55:51.854661 17271 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 07:55:51.854967 17263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0: Bootstrap starting.
I20260809 07:55:51.855614 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.846142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:37387 (local address 127.0.0.1:50834)
0809 07:55:51.846556 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:51.846571 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:51.846640 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:51.847585 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:51.847588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:51.847759 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:51.847847 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:51.847853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.849009 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.849012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:51.849790 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:51.849802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.850481 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.850485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:51.850576 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.850974 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:51.850989 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:51.852328 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:51.854071 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:51.854074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:51.854076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:51.854306 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:51.854625 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:51.854628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:51.854630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:51.854723 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:51.855166 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:51.855169 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:51.855259 (+ 90us) client_negotiation.cc:769] Sending connection context
0809 07:55:51.855356 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 07:55:51.855474 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342}
I20260809 07:55:51.856040 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:51.846986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50834 (local address 127.28.112.60:37387)
0809 07:55:51.847319 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:51.847323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:51.847335 (+ 12us) server_negotiation.cc:407] Connection header received
0809 07:55:51.847366 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:51.847369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:51.847406 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:51.847483 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:51.847960 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.848613 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.849930 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:51.850273 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:51.850364 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:51.852484 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:51.852488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:51.852490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:51.852514 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:51.853928 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.854417 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.854420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.854422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.854515 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:51.854811 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:51.854814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:51.854815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:51.855026 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:51.855119 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:51.855805 (+ 686us) server_negotiation.cc:299] Negotiation successful
0809 07:55:51.855906 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265}
I20260809 07:55:51.857331 17263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0: Neither blocks nor log segments found. Creating new log.
I20260809 07:55:51.858098 17263 log.cc:824] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:51.860540 17145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0: Bootstrap starting.
I20260809 07:55:51.862056 17263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0: No bootstrap required, opened a new log
I20260809 07:55:51.862736 17145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0: Neither blocks nor log segments found. Creating new log.
I20260809 07:55:51.863433 17145 log.cc:824] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:51.865638 17145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0: No bootstrap required, opened a new log
I20260809 07:55:51.866134 17263 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } } peers { permanent_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } } peers { permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } }
I20260809 07:55:51.866346 17263 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:55:51.866460 17263 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1b6fb21dfb3409c86ad99dafc0dc1b0, State: Initialized, Role: FOLLOWER
I20260809 07:55:51.866907 17263 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } } peers { permanent_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } } peers { permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } }
I20260809 07:55:51.868166 17263 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:55:51.868814 17145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } } peers { permanent_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } } peers { permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } }
I20260809 07:55:51.869058 17145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:55:51.869153 17145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14942c97345640798205d02047c2eae0, State: Initialized, Role: FOLLOWER
I20260809 07:55:51.869589 17145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } } peers { permanent_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } } peers { permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } }
I20260809 07:55:51.870734 17424 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } } peers { permanent_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } } peers { permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } } }
I20260809 07:55:51.870854 17424 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:55:51.870857 17145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:55:51.871178 17434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } } peers { permanent_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } } peers { permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } } }
I20260809 07:55:51.871294 17434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 07:55:51.874641 17439 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:55:51.874751 17439 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:55:51.871469 17160 master_runner.cc:186] Error getting master registration for 127.28.112.62:45507: OK, instance_id { permanent_uuid: "14942c97345640798205d02047c2eae0" instance_seqno: 1786262151384804 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 07:55:51.875151 17160 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 07:55:51.876549 17453 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:55:51.876626 17453 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260809 07:55:51.881628 17075 tablet.cc:2369] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:55:51.882141 17042 master_runner.cc:263] No leader master found from master 14942c97345640798205d02047c2eae0
I20260809 07:55:51.882205 17042 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 07:55:51.987581 17424 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:55:51.987805 17424 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } } peers { permanent_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } } peers { permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } }
I20260809 07:55:51.988956 17424 leader_election.cc:288] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14942c97345640798205d02047c2eae0 (127.28.112.62:45507), 6e72a2f5b0674ae58763e956bbeff370 (127.28.112.60:37387)
I20260809 07:55:51.989733 17105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14942c97345640798205d02047c2eae0" is_pre_election: true
I20260809 07:55:51.990077 17105 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1b6fb21dfb3409c86ad99dafc0dc1b0 in term 0.
I20260809 07:55:51.990589 17186 leader_election.cc:302] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14942c97345640798205d02047c2eae0, e1b6fb21dfb3409c86ad99dafc0dc1b0; no voters:
I20260809 07:55:51.990927 17424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:55:51.991004 17424 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:55:51.991047 17424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:55:51.992231 17424 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } } peers { permanent_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } } peers { permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } }
I20260809 07:55:51.992563 17424 leader_election.cc:288] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [CANDIDATE]: Term 1 election: Requested vote from peers 14942c97345640798205d02047c2eae0 (127.28.112.62:45507), 6e72a2f5b0674ae58763e956bbeff370 (127.28.112.60:37387)
I20260809 07:55:51.993029 17105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14942c97345640798205d02047c2eae0"
I20260809 07:55:51.993142 17105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:55:51.994369 17105 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1b6fb21dfb3409c86ad99dafc0dc1b0 in term 1.
I20260809 07:55:51.994735 17186 leader_election.cc:302] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14942c97345640798205d02047c2eae0, e1b6fb21dfb3409c86ad99dafc0dc1b0; no voters:
I20260809 07:55:51.995558 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e72a2f5b0674ae58763e956bbeff370"
I20260809 07:55:51.995802 17333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:55:51.997008 17333 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1b6fb21dfb3409c86ad99dafc0dc1b0 in term 1.
I20260809 07:55:51.997426 17334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e72a2f5b0674ae58763e956bbeff370" is_pre_election: true
I20260809 07:55:51.997568 17334 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e1b6fb21dfb3409c86ad99dafc0dc1b0 in term 1. Re-sending same reply.
I20260809 07:55:51.998238 17424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:55:51.998477 17424 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [term 1 LEADER]: Becoming Leader. State: Replica: e1b6fb21dfb3409c86ad99dafc0dc1b0, State: Running, Role: LEADER
I20260809 07:55:51.998795 17424 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } } peers { permanent_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } } peers { permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } }
I20260809 07:55:52.000681 17424 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1b6fb21dfb3409c86ad99dafc0dc1b0. Latest consensus state: current_term: 1 leader_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } } peers { permanent_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } } peers { permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } } }
I20260809 07:55:52.000795 17424 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [sys.catalog]: This master's current role is: LEADER
I20260809 07:55:52.002818 17457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:55:52.003819 17457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:55:52.008227 17334 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370 [term 1 FOLLOWER]: Refusing update from remote peer e1b6fb21dfb3409c86ad99dafc0dc1b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:55:52.008227 17105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0 [term 1 FOLLOWER]: Refusing update from remote peer e1b6fb21dfb3409c86ad99dafc0dc1b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:55:52.008898 17424 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:55:52.009425 17455 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:55:52.012660 17434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1b6fb21dfb3409c86ad99dafc0dc1b0. Latest consensus state: current_term: 1 leader_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } } peers { permanent_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } } peers { permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } } }
I20260809 07:55:52.012780 17434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:55:52.013060 17400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1b6fb21dfb3409c86ad99dafc0dc1b0. Latest consensus state: current_term: 1 leader_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } } peers { permanent_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } } peers { permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } } }
I20260809 07:55:52.013144 17400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:55:52.016283 17455 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } } peers { permanent_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } } peers { permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } } }
I20260809 07:55:52.016390 17455 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [sys.catalog]: This master's current role is: LEADER
I20260809 07:55:52.016811 17400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } } peers { permanent_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } } peers { permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } } }
I20260809 07:55:52.016916 17400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:55:52.017119 17458 mvcc.cc:205] Tried to move back new op lower bound from 7316529774619443200 to 7316529774590574592. Current Snapshot: MvccSnapshot[applied={T|T < 7316529774619443200}]
I20260809 07:55:52.017267 17464 mvcc.cc:205] Tried to move back new op lower bound from 7316529774619443200 to 7316529774590574592. Current Snapshot: MvccSnapshot[applied={T|T < 7316529774619443200}]
I20260809 07:55:52.020417 17424 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } } peers { permanent_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } } peers { permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } } }
I20260809 07:55:52.020531 17424 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0 [sys.catalog]: This master's current role is: LEADER
I20260809 07:55:52.021469 17434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14942c97345640798205d02047c2eae0" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45507 } } peers { permanent_uuid: "e1b6fb21dfb3409c86ad99dafc0dc1b0" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 39571 } } peers { permanent_uuid: "6e72a2f5b0674ae58763e956bbeff370" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 37387 } } }
I20260809 07:55:52.021584 17434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:55:52.021831 17463 mvcc.cc:205] Tried to move back new op lower bound from 7316529774619443200 to 7316529774590574592. Current Snapshot: MvccSnapshot[applied={T|T < 7316529774619443200}]
I20260809 07:55:52.023555 17457 catalog_manager.cc:1347] Generated new cluster ID: 8c52ceee307647138787ecacd4cc350f
I20260809 07:55:52.023630 17457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:55:52.057786 17457 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:55:52.059063 17457 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:55:52.072826 17457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1b6fb21dfb3409c86ad99dafc0dc1b0: Generated new TSK 0
I20260809 07:55:52.073534 17457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:55:52.853179 17416 catalog_manager.cc:1259] Loaded cluster ID: 8c52ceee307647138787ecacd4cc350f
I20260809 07:55:52.853260 17416 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370: loading cluster ID for follower catalog manager: success
I20260809 07:55:52.854964 17416 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370: acquiring CA information for follower catalog manager: success
I20260809 07:55:52.856084 17416 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6e72a2f5b0674ae58763e956bbeff370: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 07:55:52.877767 17453 catalog_manager.cc:1259] Loaded cluster ID: 8c52ceee307647138787ecacd4cc350f
I20260809 07:55:52.877843 17453 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0: loading cluster ID for follower catalog manager: success
I20260809 07:55:52.879491 17453 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0: acquiring CA information for follower catalog manager: success
I20260809 07:55:52.880579 17453 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 14942c97345640798205d02047c2eae0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 07:55:53.129060 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 17042
I20260809 07:55:53.138176 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 17160
I20260809 07:55:53.149159 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 17271
2026-08-09T07:55:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3196 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:55:53 dist-test-slave-vpvm krb5kdc[17481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:55:53 dist-test-slave-vpvm krb5kdc[17481](info): set up 2 sockets
Aug 09 07:55:53 dist-test-slave-vpvm krb5kdc[17481](info): commencing operation
krb5kdc: starting...
W20260809 07:55:54.285641 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:55:54 dist-test-slave-vpvm krb5kdc[17481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:55:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:55:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:55:54.402905 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:42685
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:43063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:42685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:55:54.510579 17498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:54.510844 17498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:54.510900 17498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:54.514822 17498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:55:54.514874 17498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:54.514900 17498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:54.514921 17498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:55:54.514943 17498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:55:54.520097 17498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:43063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:42685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:42685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:54.521119 17498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:54.522140 17498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:54.530230 17503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:54.530239 17506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:54.530295 17498 server_base.cc:1034] running on GCE node
W20260809 07:55:54.530244 17504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:54.530966 17498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:54.531961 17498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:54.533113 17498 hybrid_clock.cc:648] HybridClock initialized: now 1786262154533094 us; error 32 us; skew 500 ppm
Aug 09 07:55:54 dist-test-slave-vpvm krb5kdc[17481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262154, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:54.535717 17498 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:55:54.536527 17498 webserver.cc:460] Webserver started at http://127.28.112.62:38481/ using document root <none> and password file <none>
I20260809 07:55:54.537046 17498 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:54.537111 17498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:54.537297 17498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:55:54.539285 17498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "0a8a5285de0a4771ab92b995d5cb55fc"
format_stamp: "Formatted at 2026-08-09 07:55:54 on dist-test-slave-vpvm"
server_key: "b4f7ccda674a7ad324e95462e59ad738"
server_key_iv: "ee4b76ce8236a58941b9d57f21be5bcb"
server_key_version: "encryptionkey@0"
I20260809 07:55:54.539821 17498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "0a8a5285de0a4771ab92b995d5cb55fc"
format_stamp: "Formatted at 2026-08-09 07:55:54 on dist-test-slave-vpvm"
server_key: "b4f7ccda674a7ad324e95462e59ad738"
server_key_iv: "ee4b76ce8236a58941b9d57f21be5bcb"
server_key_version: "encryptionkey@0"
I20260809 07:55:54.543311 17498 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 07:55:54.545519 17518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:54.546663 17498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:55:54.546777 17498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "0a8a5285de0a4771ab92b995d5cb55fc"
format_stamp: "Formatted at 2026-08-09 07:55:54 on dist-test-slave-vpvm"
server_key: "b4f7ccda674a7ad324e95462e59ad738"
server_key_iv: "ee4b76ce8236a58941b9d57f21be5bcb"
server_key_version: "encryptionkey@0"
I20260809 07:55:54.546866 17498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:54.570947 17498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:54.573968 17498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:54.574151 17498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:54.580976 17498 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:42685
I20260809 07:55:54.580991 17611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:42685 every 8 connection(s)
I20260809 07:55:54.581890 17498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:55:54.584690 17612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:55:54.589047 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 17498
I20260809 07:55:54.589169 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:55:54.589411 29120 external_mini_cluster.cc:1428] Setting key 9edde6f04d6050f90ec37e48cfb0fd12
I20260809 07:55:54.590240 17612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc: Bootstrap starting.
I20260809 07:55:54.592808 17612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc: Neither blocks nor log segments found. Creating new log.
I20260809 07:55:54.593663 17612 log.cc:824] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:54.595659 17612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc: No bootstrap required, opened a new log
Aug 09 07:55:54 dist-test-slave-vpvm krb5kdc[17481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:55:54.599718 17612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8a5285de0a4771ab92b995d5cb55fc" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 42685 } }
I20260809 07:55:54.600009 17612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:55:54.600075 17612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a8a5285de0a4771ab92b995d5cb55fc, State: Initialized, Role: FOLLOWER
I20260809 07:55:54.600562 17612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8a5285de0a4771ab92b995d5cb55fc" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 42685 } }
I20260809 07:55:54.600726 17612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:55:54.600793 17612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:55:54.600891 17612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:55:54.601908 17612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8a5285de0a4771ab92b995d5cb55fc" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 42685 } }
I20260809 07:55:54.602294 17612 leader_election.cc:302] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a8a5285de0a4771ab92b995d5cb55fc; no voters:
I20260809 07:55:54.602630 17612 leader_election.cc:288] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:55:54.602986 17622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:55:54.603197 17622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc [term 1 LEADER]: Becoming Leader. State: Replica: 0a8a5285de0a4771ab92b995d5cb55fc, State: Running, Role: LEADER
I20260809 07:55:54.603541 17622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8a5285de0a4771ab92b995d5cb55fc" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 42685 } }
I20260809 07:55:54.603914 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:54.590698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.28.112.62:42685)
0809 07:55:54.591111 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:54.591130 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:54.591171 (+ 41us) server_negotiation.cc:407] Connection header received
0809 07:55:54.591979 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:54.592002 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:54.592394 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:54.592751 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:54.593437 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:54.594772 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:54.595615 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:54.596268 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:54.596436 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:54.598976 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:54.598983 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:54.599000 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:54.599031 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:54.600850 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:54.601282 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:54.601288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:54.601294 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:54.601428 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:54.601681 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:54.601685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:54.601686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:54.602243 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:54.602477 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:54.602678 (+ 201us) server_negotiation.cc:299] Negotiation successful
0809 07:55:54.602981 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":108,"threads_started":1}
I20260809 07:55:54.603919 17612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:55:54.604715 17620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a8a5285de0a4771ab92b995d5cb55fc. Latest consensus state: current_term: 1 leader_uuid: "0a8a5285de0a4771ab92b995d5cb55fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8a5285de0a4771ab92b995d5cb55fc" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 42685 } } }
I20260809 07:55:54.604835 17620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc [sys.catalog]: This master's current role is: LEADER
I20260809 07:55:54.605578 17623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a8a5285de0a4771ab92b995d5cb55fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8a5285de0a4771ab92b995d5cb55fc" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 42685 } } }
I20260809 07:55:54.605731 17623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc [sys.catalog]: This master's current role is: LEADER
I20260809 07:55:54.607306 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260809 07:55:54.609059 17644 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:55:54.609161 17644 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:55:54.609237 17625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:55:54.610205 17625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:55:54.615309 17625 catalog_manager.cc:1347] Generated new cluster ID: f36566e49c3d474bbe8f5dd6f0f04c89
I20260809 07:55:54.615372 17625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:55:54.653414 17625 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:55:54.654799 17625 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:55:54.659145 17625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a8a5285de0a4771ab92b995d5cb55fc: Generated new TSK 0
I20260809 07:55:54.659917 17625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:55:54.714392 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 17498
2026-08-09T07:55:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1567 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:55:54 dist-test-slave-vpvm krb5kdc[17657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:55:54 dist-test-slave-vpvm krb5kdc[17657](info): set up 2 sockets
Aug 09 07:55:54 dist-test-slave-vpvm krb5kdc[17657](info): commencing operation
krb5kdc: starting...
W20260809 07:55:55.862823 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:55:55 dist-test-slave-vpvm krb5kdc[17657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:55:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:55:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:55:55.982298 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:43185
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:43185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:55:56.091123 17671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:56.091346 17671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:56.091383 17671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:56.095227 17671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:55:56.095273 17671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:56.095289 17671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:56.095304 17671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:55:56.095321 17671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:55:56.100314 17671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:43185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:43185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:56.101215 17671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:56.102198 17671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:56.110117 17678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:56.110208 17671 server_base.cc:1034] running on GCE node
W20260809 07:55:56.110145 17683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:56.110096 17680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:56.110846 17671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:56.111832 17671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:56.112989 17671 hybrid_clock.cc:648] HybridClock initialized: now 1786262156112975 us; error 25 us; skew 500 ppm
Aug 09 07:55:56 dist-test-slave-vpvm krb5kdc[17657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262156, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:56.115572 17671 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:55:56.116354 17671 webserver.cc:460] Webserver started at http://127.28.112.62:46003/ using document root <none> and password file <none>
I20260809 07:55:56.116850 17671 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:56.116912 17671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:56.117087 17671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:55:56.118993 17671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "778a7b6b0e6a40118c54ad5712e7771d"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "1baa5122156ccaff2315d975fe33a374"
server_key_iv: "ebd99b78a01fbf91592a3ca784b42cc7"
server_key_version: "encryptionkey@0"
I20260809 07:55:56.119511 17671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "778a7b6b0e6a40118c54ad5712e7771d"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "1baa5122156ccaff2315d975fe33a374"
server_key_iv: "ebd99b78a01fbf91592a3ca784b42cc7"
server_key_version: "encryptionkey@0"
I20260809 07:55:56.122880 17671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:55:56.125059 17694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:56.126192 17671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:55:56.126304 17671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "778a7b6b0e6a40118c54ad5712e7771d"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "1baa5122156ccaff2315d975fe33a374"
server_key_iv: "ebd99b78a01fbf91592a3ca784b42cc7"
server_key_version: "encryptionkey@0"
I20260809 07:55:56.126389 17671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:56.154155 17671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:56.156726 17671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:56.156898 17671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:56.163437 17671 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:43185
I20260809 07:55:56.163714 17770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:43185 every 8 connection(s)
I20260809 07:55:56.164350 17671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:55:56.167049 17774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:55:56.167929 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 17671
I20260809 07:55:56.168041 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:55:56.168315 29120 external_mini_cluster.cc:1428] Setting key 31807b083f46e0d5093ff35fd419895e
I20260809 07:55:56.173014 17774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d: Bootstrap starting.
I20260809 07:55:56.175596 17774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d: Neither blocks nor log segments found. Creating new log.
I20260809 07:55:56.176499 17774 log.cc:824] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d: Log is configured to *not* fsync() on all Append() calls
Aug 09 07:55:56 dist-test-slave-vpvm krb5kdc[17657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:55:56.178303 17774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d: No bootstrap required, opened a new log
I20260809 07:55:56.181615 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:56.169575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57350 (local address 127.28.112.62:43185)
0809 07:55:56.169999 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:56.170016 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:56.170051 (+ 35us) server_negotiation.cc:407] Connection header received
0809 07:55:56.170661 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:56.170677 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:56.171020 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:56.171279 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:56.172176 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.173106 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.174268 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.174707 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.174825 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:56.177429 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:56.177435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:56.177447 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:56.177474 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:56.179014 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:56.179466 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:56.179472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:56.179477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:56.179582 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:56.179834 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:56.179838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:56.179840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:56.180215 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:56.180389 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:56.180672 (+ 283us) server_negotiation.cc:299] Negotiation successful
0809 07:55:56.180944 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":106,"threads_started":1}
I20260809 07:55:56.181615 17774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a7b6b0e6a40118c54ad5712e7771d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43185 } }
I20260809 07:55:56.181912 17774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:55:56.181994 17774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778a7b6b0e6a40118c54ad5712e7771d, State: Initialized, Role: FOLLOWER
I20260809 07:55:56.182466 17774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a7b6b0e6a40118c54ad5712e7771d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43185 } }
I20260809 07:55:56.182622 17774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:55:56.182688 17774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:55:56.182782 17774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:55:56.183583 17774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a7b6b0e6a40118c54ad5712e7771d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43185 } }
I20260809 07:55:56.183940 17774 leader_election.cc:302] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 778a7b6b0e6a40118c54ad5712e7771d; no voters:
I20260809 07:55:56.184216 17774 leader_election.cc:288] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:55:56.184301 17779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:55:56.184536 17779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 1 LEADER]: Becoming Leader. State: Replica: 778a7b6b0e6a40118c54ad5712e7771d, State: Running, Role: LEADER
I20260809 07:55:56.184854 17779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a7b6b0e6a40118c54ad5712e7771d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43185 } }
I20260809 07:55:56.185369 17774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:55:56.186574 17780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "778a7b6b0e6a40118c54ad5712e7771d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a7b6b0e6a40118c54ad5712e7771d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43185 } } }
I20260809 07:55:56.186604 17781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 778a7b6b0e6a40118c54ad5712e7771d. Latest consensus state: current_term: 1 leader_uuid: "778a7b6b0e6a40118c54ad5712e7771d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a7b6b0e6a40118c54ad5712e7771d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43185 } } }
I20260809 07:55:56.186759 17781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [sys.catalog]: This master's current role is: LEADER
I20260809 07:55:56.187023 17780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [sys.catalog]: This master's current role is: LEADER
I20260809 07:55:56.187161 17795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:55:56.190430 17795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:55:56.195052 17795 catalog_manager.cc:1347] Generated new cluster ID: b05d8f3e7cde437daca2bc708fc91895
I20260809 07:55:56.195114 17795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:55:56.199643 17795 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:55:56.200430 17795 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:55:56.204223 17795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d: Generated new TSK 0
I20260809 07:55:56.204770 17795 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
I20260809 07:55:56.233004 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:43185
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:55:56.344551 17812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:56.344796 17812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:56.344836 17812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:56.348707 17812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:56.348759 17812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:56.348814 17812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:55:56.353729 17812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:56.354722 17812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:56.355671 17812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:56.364805 17821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:56.364897 17812 server_base.cc:1034] running on GCE node
W20260809 07:55:56.364800 17819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:56.364800 17818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:56.365473 17812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:56.366056 17812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:56.367210 17812 hybrid_clock.cc:648] HybridClock initialized: now 1786262156367194 us; error 27 us; skew 500 ppm
Aug 09 07:55:56 dist-test-slave-vpvm krb5kdc[17657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262156, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:56.369817 17812 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:55:56.370639 17812 webserver.cc:460] Webserver started at http://127.28.112.1:39571/ using document root <none> and password file <none>
I20260809 07:55:56.371171 17812 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:56.371238 17812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:56.371400 17812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:55:56.373297 17812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "1fbc44e8df044db58a770554ee292de6"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "a6b45360ad9baaeb79d15e2c8b0008f1"
server_key_iv: "3a7c6f36b4cedb2974a1e3f5a4063d97"
server_key_version: "encryptionkey@0"
I20260809 07:55:56.373780 17812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "1fbc44e8df044db58a770554ee292de6"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "a6b45360ad9baaeb79d15e2c8b0008f1"
server_key_iv: "3a7c6f36b4cedb2974a1e3f5a4063d97"
server_key_version: "encryptionkey@0"
I20260809 07:55:56.377161 17812 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 07:55:56.379381 17830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:56.380514 17812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:55:56.380609 17812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "1fbc44e8df044db58a770554ee292de6"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "a6b45360ad9baaeb79d15e2c8b0008f1"
server_key_iv: "3a7c6f36b4cedb2974a1e3f5a4063d97"
server_key_version: "encryptionkey@0"
I20260809 07:55:56.380686 17812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:56.392688 17812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:56.395105 17812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:56.395236 17812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:56.395692 17812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:55:56.396401 17812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:55:56.396447 17812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:56.396481 17812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:55:56.396502 17812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:56.404407 17812 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:35403
I20260809 07:55:56.404450 18008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:35403 every 8 connection(s)
I20260809 07:55:56.405205 17812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:55:56.408859 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 17812
I20260809 07:55:56.408985 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:55:56.409250 29120 external_mini_cluster.cc:1428] Setting key 8c9e794a87b180c153fb7406a12a22db
Aug 09 07:55:56 dist-test-slave-vpvm krb5kdc[17657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262156, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:55:56.418836 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:56.406891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:34049 (local address 127.28.112.62:43185)
0809 07:55:56.407034 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:56.407038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:56.407850 (+ 812us) server_negotiation.cc:407] Connection header received
0809 07:55:56.409212 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:56.409216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:56.409282 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:56.409351 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:56.410662 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.411220 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.412708 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.413044 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.413152 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:56.415552 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:56.415555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:56.415557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:56.415599 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:56.416841 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:56.417393 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:56.417396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:56.417397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:56.417461 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:56.417797 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:56.417799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:56.417800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:56.417981 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:56.418057 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:56.418620 (+ 563us) server_negotiation.cc:299] Negotiation successful
0809 07:55:56.418719 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.28.112.2@KRBTEST.COM; defaulting to no policy
I20260809 07:55:56.419870 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:56.407212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:34049)
0809 07:55:56.407696 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:56.407736 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:56.408991 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:56.409553 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:56.409573 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:56.409910 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:56.410519 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:56.410536 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.411345 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.411348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:56.412572 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:56.412585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.413176 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.413178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:56.413311 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:56.413809 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:56.413824 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:56.415377 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:56.416955 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:56.416962 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:56.416977 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:56.417304 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:56.417537 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:56.417540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:56.417542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:56.417715 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:56.418070 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:56.418077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:56.418366 (+ 289us) client_negotiation.cc:769] Sending connection context
0809 07:55:56.418589 (+ 223us) client_negotiation.cc:240] Negotiation successful
0809 07:55:56.418910 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":112,"threads_started":1}
I20260809 07:55:56.420956 18009 heartbeater.cc:347] Connected to a master server at 127.28.112.62:43185
I20260809 07:55:56.421172 18009 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:56.421659 18009 heartbeater.cc:511] Master 127.28.112.62:43185 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.2@KRBTEST.COM" created.
I20260809 07:55:56.423393 17716 ts_manager.cc:194] Registered new tserver with Master: 1fbc44e8df044db58a770554ee292de6 (127.28.112.1:35403)
I20260809 07:55:56.424539 17716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:34049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
I20260809 07:55:56.435969 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:43185
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:55:56.542934 18019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:56.543195 18019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:56.543254 18019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:56.547147 18019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:56.547204 18019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:56.547279 18019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:55:56.552349 18019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:56.553355 18019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:56.554390 18019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:56.563796 18027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:56.563858 18019 server_base.cc:1034] running on GCE node
W20260809 07:55:56.563807 18025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:56.563830 18024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:56.564342 18019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:56.564949 18019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:56.566100 18019 hybrid_clock.cc:648] HybridClock initialized: now 1786262156566076 us; error 35 us; skew 500 ppm
Aug 09 07:55:56 dist-test-slave-vpvm krb5kdc[17657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262156, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:56.568706 18019 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:55:56.569501 18019 webserver.cc:460] Webserver started at http://127.28.112.2:41211/ using document root <none> and password file <none>
I20260809 07:55:56.570034 18019 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:56.570098 18019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:56.570274 18019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:55:56.572325 18019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "779e66f2e58a4f6ab3e310b66dbade52"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "3e8e3533041c0b25cc8dab5779a791ad"
server_key_iv: "39117c70ec5ec17281ea145b61e411e2"
server_key_version: "encryptionkey@0"
I20260809 07:55:56.572831 18019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "779e66f2e58a4f6ab3e310b66dbade52"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "3e8e3533041c0b25cc8dab5779a791ad"
server_key_iv: "39117c70ec5ec17281ea145b61e411e2"
server_key_version: "encryptionkey@0"
I20260809 07:55:56.576323 18019 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 07:55:56.578586 18040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:56.579702 18019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:55:56.579799 18019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "779e66f2e58a4f6ab3e310b66dbade52"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "3e8e3533041c0b25cc8dab5779a791ad"
server_key_iv: "39117c70ec5ec17281ea145b61e411e2"
server_key_version: "encryptionkey@0"
I20260809 07:55:56.579895 18019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:56.605017 18019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:56.607803 18019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:56.607970 18019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:56.608523 18019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:55:56.609334 18019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:55:56.609382 18019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:56.609422 18019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:55:56.609436 18019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:56.617703 18019 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:32773
I20260809 07:55:56.617733 18215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:32773 every 8 connection(s)
I20260809 07:55:56.618533 18019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:55:56.621601 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 18019
I20260809 07:55:56.621709 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:55:56.621981 29120 external_mini_cluster.cc:1428] Setting key 14a41f192e36210fe6a7817d538dbb87
Aug 09 07:55:56 dist-test-slave-vpvm krb5kdc[17657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262156, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.112.3@KRBTEST.COM; defaulting to no policy
I20260809 07:55:56.632316 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:56.620228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:35161 (local address 127.28.112.62:43185)
0809 07:55:56.620369 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:56.620374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:56.621185 (+ 811us) server_negotiation.cc:407] Connection header received
0809 07:55:56.622605 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:56.622608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:56.622671 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:56.622756 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:56.624097 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.624835 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.626278 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.626495 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.626573 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:56.629088 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:56.629091 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:56.629093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:56.629117 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:56.630431 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:56.630911 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:56.630914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:56.630915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:56.630990 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:56.631305 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:56.631309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:56.631310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:56.631484 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:56.631569 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:56.632035 (+ 466us) server_negotiation.cc:299] Negotiation successful
0809 07:55:56.632154 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 07:55:56.633068 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:56.620586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:35161)
0809 07:55:56.621041 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:56.621083 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:56.622368 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:56.622944 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:56.622953 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:56.623476 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:56.623953 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:56.623967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.624957 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.624960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:56.626165 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:56.626177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.626638 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.626642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:56.626831 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:56.627373 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:56.627392 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:56.628931 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:56.630555 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:56.630561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:56.630575 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:56.630816 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:56.631072 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:56.631076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:56.631078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:56.631216 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:56.631585 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:56.631592 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:56.631814 (+ 222us) client_negotiation.cc:769] Sending connection context
0809 07:55:56.631977 (+ 163us) client_negotiation.cc:240] Negotiation successful
0809 07:55:56.632294 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":108,"threads_started":1}
I20260809 07:55:56.634169 18220 heartbeater.cc:347] Connected to a master server at 127.28.112.62:43185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.3@KRBTEST.COM" created.
I20260809 07:55:56.634415 18220 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:56.634956 18220 heartbeater.cc:511] Master 127.28.112.62:43185 requested a full tablet report, sending...
I20260809 07:55:56.636082 17716 ts_manager.cc:194] Registered new tserver with Master: 779e66f2e58a4f6ab3e310b66dbade52 (127.28.112.2:32773)
I20260809 07:55:56.636818 17716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:35161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
I20260809 07:55:56.648888 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:43185
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:55:56.755841 18231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:56.756074 18231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:56.756132 18231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:56.760061 18231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:56.760113 18231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:56.760186 18231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:55:56.765115 18231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:56.766007 18231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:56.766888 18231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:56.775725 18244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:56.775760 18239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:56.775701 18238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:56.775856 18231 server_base.cc:1034] running on GCE node
I20260809 07:55:56.776302 18231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:56.776883 18231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:56.778018 18231 hybrid_clock.cc:648] HybridClock initialized: now 1786262156778001 us; error 25 us; skew 500 ppm
Aug 09 07:55:56 dist-test-slave-vpvm krb5kdc[17657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262156, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:56.780608 18231 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:55:56.781358 18231 webserver.cc:460] Webserver started at http://127.28.112.3:37685/ using document root <none> and password file <none>
I20260809 07:55:56.781853 18231 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:56.781916 18231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:56.782120 18231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:55:56.784145 18231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "7113fd4151764215a0c7cada8f56ebcc"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "f1e26a4af2be627a4e02bce7713d2255"
server_key_iv: "698ebd02c689894bb1a128163f589a70"
server_key_version: "encryptionkey@0"
I20260809 07:55:56.784682 18231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "7113fd4151764215a0c7cada8f56ebcc"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "f1e26a4af2be627a4e02bce7713d2255"
server_key_iv: "698ebd02c689894bb1a128163f589a70"
server_key_version: "encryptionkey@0"
I20260809 07:55:56.788240 18231 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 07:55:56.790500 18256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:56.791612 18231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:55:56.791734 18231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "7113fd4151764215a0c7cada8f56ebcc"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "f1e26a4af2be627a4e02bce7713d2255"
server_key_iv: "698ebd02c689894bb1a128163f589a70"
server_key_version: "encryptionkey@0"
I20260809 07:55:56.791827 18231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:56.810881 18231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:56.813886 18231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:56.814081 18231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:56.814666 18231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:55:56.815510 18231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:55:56.815559 18231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:56.815613 18231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:55:56.815649 18231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:56.824160 18231 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:39997
I20260809 07:55:56.824214 18419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:39997 every 8 connection(s)
I20260809 07:55:56.825011 18231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
Aug 09 07:55:56 dist-test-slave-vpvm krb5kdc[17657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262156, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:55:56.834167 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 18231
I20260809 07:55:56.834275 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:55:56.834576 29120 external_mini_cluster.cc:1428] Setting key dbc84060d8944850642896cd5b17087f
I20260809 07:55:56.837553 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:56.826717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:44879 (local address 127.28.112.62:43185)
0809 07:55:56.826844 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:56.826848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:56.827630 (+ 782us) server_negotiation.cc:407] Connection header received
0809 07:55:56.828932 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:56.828936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:56.828993 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:56.829067 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:56.830225 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.830773 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.831857 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.832074 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.832162 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:56.834371 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:56.834375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:56.834378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:56.834409 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:56.835794 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:56.836255 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:56.836259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:56.836261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:56.836353 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:56.836706 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:56.836709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:56.836710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:56.836887 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:56.836965 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:56.837349 (+ 384us) server_negotiation.cc:299] Negotiation successful
0809 07:55:56.837440 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 07:55:56.838316 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:56.827053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:44879)
0809 07:55:56.827493 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:56.827533 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:56.828715 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:56.829236 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:56.829243 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:56.829587 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:56.830089 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:56.830105 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.830891 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.830895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:56.831733 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:56.831742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.832210 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.832214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:56.832372 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:56.832823 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:56.832838 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:56.834205 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:56.835898 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:56.835904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:56.835918 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:56.836163 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:56.836463 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:56.836466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:56.836468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:56.836619 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:56.836964 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:56.836968 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:56.837174 (+ 206us) client_negotiation.cc:769] Sending connection context
0809 07:55:56.837316 (+ 142us) client_negotiation.cc:240] Negotiation successful
0809 07:55:56.837571 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":115,"threads_started":1}
I20260809 07:55:56.839285 18420 heartbeater.cc:347] Connected to a master server at 127.28.112.62:43185
I20260809 07:55:56.839481 18420 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:56.839884 18420 heartbeater.cc:511] Master 127.28.112.62:43185 requested a full tablet report, sending...
I20260809 07:55:56.840772 17716 ts_manager.cc:194] Registered new tserver with Master: 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997)
I20260809 07:55:56.841313 17716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:44879
I20260809 07:55:56.848089 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 07:55:56.856383 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:56.849871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.28.112.62:43185)
0809 07:55:56.850016 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:56.850020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:56.850184 (+ 164us) server_negotiation.cc:407] Connection header received
0809 07:55:56.850299 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:56.850302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:56.850345 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:56.850405 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:56.851047 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.851532 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.852325 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.852537 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.852620 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:56.853762 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:56.853766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:56.853769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:56.853793 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:56.854890 (+ 1097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:56.855278 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:56.855281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:56.855283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:56.855355 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:56.855594 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:56.855598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:56.855599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:56.855807 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:56.855920 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:56.856143 (+ 223us) server_negotiation.cc:299] Negotiation successful
0809 07:55:56.856244 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 07:55:56.858808 17716 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:55:56.861806 17716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:55:56.875821 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:56.870707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58434 (local address 127.28.112.3:39997)
0809 07:55:56.871221 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:56.871232 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:56.871249 (+ 17us) server_negotiation.cc:407] Connection header received
0809 07:55:56.871308 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:56.871312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:56.871464 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:56.872126 (+ 662us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:55:56.872262 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.873328 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.874548 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.875182 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.875280 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:56.875423 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:56.875595 (+ 172us) server_negotiation.cc:299] Negotiation successful
0809 07:55:56.875684 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":156,"threads_started":1}
I20260809 07:55:56.875813 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:56.870258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:39997 (local address 127.0.0.1:58434)
0809 07:55:56.870654 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:56.870702 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:56.870838 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:56.871608 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:56.871614 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:56.871654 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:55:56.871799 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:56.871806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.873433 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.873436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:56.874420 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:56.874429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.875295 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.875299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:56.875402 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:56.875452 (+ 50us) client_negotiation.cc:769] Sending connection context
0809 07:55:56.875553 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 07:55:56.875667 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":29,"spinlock_wait_cycles":18048,"thread_start_us":77,"threads_started":1}
I20260809 07:55:56.876147 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:56.869892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:32773 (local address 127.0.0.1:39680)
0809 07:55:56.870649 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:56.870700 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:56.870838 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:56.871608 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:56.871614 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:56.871658 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:55:56.871965 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:56.871972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.873363 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.873366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:56.874666 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:56.874678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.875770 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.875774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:56.875862 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:56.875885 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 07:55:56.875934 (+ 49us) client_negotiation.cc:240] Negotiation successful
0809 07:55:56.875998 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":278,"spinlock_wait_cycles":50048,"thread_start_us":101,"threads_started":1}
I20260809 07:55:56.876447 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:56.871058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.28.112.1:35403)
0809 07:55:56.871376 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:56.871387 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:56.871754 (+ 367us) server_negotiation.cc:407] Connection header received
0809 07:55:56.871835 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:56.871840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:56.872021 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:56.872172 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:55:56.872382 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.873077 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.874815 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.875628 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.875736 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:56.875870 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:56.876221 (+ 351us) server_negotiation.cc:299] Negotiation successful
0809 07:55:56.876312 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":129,"threads_started":1}
I20260809 07:55:56.876482 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:56.870644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:35403 (local address 127.0.0.1:33004)
0809 07:55:56.871547 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:56.871559 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:56.871655 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:56.872163 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:56.872166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:56.872185 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:55:56.872280 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:56.872286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.873200 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.873203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:56.874700 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:56.874708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.875929 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.875932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:56.876007 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:56.876026 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 07:55:56.876087 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 07:55:56.876145 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":89,"spinlock_wait_cycles":7936,"thread_start_us":95,"threads_started":1}
I20260809 07:55:56.877053 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:56.870091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.28.112.2:32773)
0809 07:55:56.871121 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:56.871133 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:56.871155 (+ 22us) server_negotiation.cc:407] Connection header received
0809 07:55:56.871231 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:56.871237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:56.871433 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:56.871734 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:55:56.872059 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.873241 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.874813 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:56.875628 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:56.876652 (+ 1024us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:56.876788 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:56.876834 (+ 46us) server_negotiation.cc:299] Negotiation successful
0809 07:55:56.876917 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925,"thread_start_us":233,"threads_started":1}
I20260809 07:55:56.879206 17913 tablet_service.cc:1467] Processing CreateTablet for tablet a16c64a9eb1c453c8277a3eff5b72896 (DEFAULT_TABLE table=test-table [id=caf3c7f8ac404e4ebc56e32601df2b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:55:56.879215 18323 tablet_service.cc:1467] Processing CreateTablet for tablet a16c64a9eb1c453c8277a3eff5b72896 (DEFAULT_TABLE table=test-table [id=caf3c7f8ac404e4ebc56e32601df2b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:55:56.879522 18108 tablet_service.cc:1467] Processing CreateTablet for tablet a16c64a9eb1c453c8277a3eff5b72896 (DEFAULT_TABLE table=test-table [id=caf3c7f8ac404e4ebc56e32601df2b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:55:56.880196 17913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a16c64a9eb1c453c8277a3eff5b72896. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:55:56.880240 18323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a16c64a9eb1c453c8277a3eff5b72896. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:55:56.880378 18108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a16c64a9eb1c453c8277a3eff5b72896. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:55:56.886914 18447 tablet_bootstrap.cc:492] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6: Bootstrap starting.
I20260809 07:55:56.888175 18448 tablet_bootstrap.cc:492] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52: Bootstrap starting.
I20260809 07:55:56.888581 18449 tablet_bootstrap.cc:492] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc: Bootstrap starting.
I20260809 07:55:56.889541 18447 tablet_bootstrap.cc:654] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6: Neither blocks nor log segments found. Creating new log.
I20260809 07:55:56.890321 18449 tablet_bootstrap.cc:654] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc: Neither blocks nor log segments found. Creating new log.
I20260809 07:55:56.890710 18448 tablet_bootstrap.cc:654] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52: Neither blocks nor log segments found. Creating new log.
I20260809 07:55:56.890969 18447 log.cc:824] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:56.891338 18449 log.cc:824] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:56.891517 18448 log.cc:824] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:56.893131 18449 tablet_bootstrap.cc:492] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc: No bootstrap required, opened a new log
I20260809 07:55:56.893285 18447 tablet_bootstrap.cc:492] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6: No bootstrap required, opened a new log
I20260809 07:55:56.893285 18448 tablet_bootstrap.cc:492] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52: No bootstrap required, opened a new log
I20260809 07:55:56.893311 18449 ts_tablet_manager.cc:1397] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 07:55:56.893472 18448 ts_tablet_manager.cc:1397] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 07:55:56.893472 18447 ts_tablet_manager.cc:1397] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260809 07:55:56.896356 18449 raft_consensus.cc:348] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } }
I20260809 07:55:56.896590 18449 raft_consensus.cc:374] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:55:56.896663 18449 raft_consensus.cc:729] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7113fd4151764215a0c7cada8f56ebcc, State: Initialized, Role: FOLLOWER
I20260809 07:55:56.897087 18449 consensus_queue.cc:260] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } }
I20260809 07:55:56.897233 18447 raft_consensus.cc:348] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } }
I20260809 07:55:56.897233 18448 raft_consensus.cc:348] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } }
I20260809 07:55:56.897480 18448 raft_consensus.cc:374] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:55:56.897549 18448 raft_consensus.cc:729] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 779e66f2e58a4f6ab3e310b66dbade52, State: Initialized, Role: FOLLOWER
I20260809 07:55:56.897732 18447 raft_consensus.cc:374] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:55:56.897770 18420 heartbeater.cc:503] Master 127.28.112.62:43185 was elected leader, sending a full tablet report...
I20260809 07:55:56.897830 18447 raft_consensus.cc:729] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fbc44e8df044db58a770554ee292de6, State: Initialized, Role: FOLLOWER
I20260809 07:55:56.898114 18449 ts_tablet_manager.cc:1428] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 07:55:56.898039 18448 consensus_queue.cc:260] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } }
I20260809 07:55:56.898275 18447 consensus_queue.cc:260] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } }
I20260809 07:55:56.898829 18220 heartbeater.cc:503] Master 127.28.112.62:43185 was elected leader, sending a full tablet report...
I20260809 07:55:56.899375 18447 ts_tablet_manager.cc:1428] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 07:55:56.899753 18009 heartbeater.cc:503] Master 127.28.112.62:43185 was elected leader, sending a full tablet report...
I20260809 07:55:56.899999 18448 ts_tablet_manager.cc:1428] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260809 07:55:56.906185 18013 tablet.cc:2369] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:55:57.025063 18454 raft_consensus.cc:482] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:55:57.025285 18454 raft_consensus.cc:504] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } }
I20260809 07:55:57.026486 18454 leader_election.cc:288] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fbc44e8df044db58a770554ee292de6 (127.28.112.1:35403), 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997)
I20260809 07:55:57.029888 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.026851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:51757 (local address 127.28.112.3:39997)
0809 07:55:57.026973 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:57.026978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:57.026993 (+ 15us) server_negotiation.cc:407] Connection header received
0809 07:55:57.027099 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:57.027102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:57.027141 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:57.027203 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:55:57.027464 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.027957 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.028932 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.029369 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.029481 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:57.029556 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:57.029708 (+ 152us) server_negotiation.cc:299] Negotiation successful
0809 07:55:57.029776 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 07:55:57.029888 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.026722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:39997 (local address 127.28.112.2:51757)
0809 07:55:57.026856 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:57.026872 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:57.026989 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:57.027224 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:57.027227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:57.027246 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:55:57.027355 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:57.027360 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.028055 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.028059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:57.028814 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:57.028822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.029490 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.029494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:57.029598 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:57.029619 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 07:55:57.029684 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 07:55:57.029755 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260809 07:55:57.030290 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.026829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:55499 (local address 127.28.112.1:35403)
0809 07:55:57.026981 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:57.026986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:57.027363 (+ 377us) server_negotiation.cc:407] Connection header received
0809 07:55:57.027481 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:57.027485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:57.027528 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:57.027590 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:55:57.027842 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.028309 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.029306 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.029712 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.029823 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:57.029875 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:57.030092 (+ 217us) server_negotiation.cc:299] Negotiation successful
0809 07:55:57.030173 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 07:55:57.030572 18348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16c64a9eb1c453c8277a3eff5b72896" candidate_uuid: "779e66f2e58a4f6ab3e310b66dbade52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7113fd4151764215a0c7cada8f56ebcc" is_pre_election: true
I20260809 07:55:57.030622 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.026722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:35403 (local address 127.28.112.2:55499)
0809 07:55:57.027267 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:57.027279 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:57.027381 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:57.027610 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:57.027613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:57.027628 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:55:57.027740 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:57.027746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.028409 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.028412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:57.029190 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:57.029200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.029824 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.029827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:57.029922 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:57.029943 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 07:55:57.030435 (+ 492us) client_negotiation.cc:240] Negotiation successful
0809 07:55:57.030510 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":176,"thread_start_us":139,"threads_started":1}
I20260809 07:55:57.030880 18348 raft_consensus.cc:2454] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 779e66f2e58a4f6ab3e310b66dbade52 in term 0.
I20260809 07:55:57.031221 17945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16c64a9eb1c453c8277a3eff5b72896" candidate_uuid: "779e66f2e58a4f6ab3e310b66dbade52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fbc44e8df044db58a770554ee292de6" is_pre_election: true
I20260809 07:55:57.031347 18048 leader_election.cc:302] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7113fd4151764215a0c7cada8f56ebcc, 779e66f2e58a4f6ab3e310b66dbade52; no voters:
I20260809 07:55:57.031468 17945 raft_consensus.cc:2454] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 779e66f2e58a4f6ab3e310b66dbade52 in term 0.
I20260809 07:55:57.031647 18454 raft_consensus.cc:2781] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:55:57.031733 18454 raft_consensus.cc:482] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:55:57.031790 18454 raft_consensus.cc:3037] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:55:57.032785 18454 raft_consensus.cc:504] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } }
I20260809 07:55:57.033073 18454 leader_election.cc:288] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [CANDIDATE]: Term 1 election: Requested vote from peers 1fbc44e8df044db58a770554ee292de6 (127.28.112.1:35403), 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997)
I20260809 07:55:57.033406 17945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16c64a9eb1c453c8277a3eff5b72896" candidate_uuid: "779e66f2e58a4f6ab3e310b66dbade52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fbc44e8df044db58a770554ee292de6"
I20260809 07:55:57.033432 18348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a16c64a9eb1c453c8277a3eff5b72896" candidate_uuid: "779e66f2e58a4f6ab3e310b66dbade52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7113fd4151764215a0c7cada8f56ebcc"
I20260809 07:55:57.033514 17945 raft_consensus.cc:3037] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:55:57.033524 18348 raft_consensus.cc:3037] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:55:57.034497 18348 raft_consensus.cc:2454] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 779e66f2e58a4f6ab3e310b66dbade52 in term 1.
I20260809 07:55:57.034584 17945 raft_consensus.cc:2454] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 779e66f2e58a4f6ab3e310b66dbade52 in term 1.
I20260809 07:55:57.034806 18048 leader_election.cc:302] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7113fd4151764215a0c7cada8f56ebcc, 779e66f2e58a4f6ab3e310b66dbade52; no voters:
I20260809 07:55:57.034984 18454 raft_consensus.cc:2781] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:55:57.035182 18454 raft_consensus.cc:686] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [term 1 LEADER]: Becoming Leader. State: Replica: 779e66f2e58a4f6ab3e310b66dbade52, State: Running, Role: LEADER
I20260809 07:55:57.035470 18454 consensus_queue.cc:237] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } }
I20260809 07:55:57.038398 17714 catalog_manager.cc:5495] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 779e66f2e58a4f6ab3e310b66dbade52 (127.28.112.2). New cstate: current_term: 1 leader_uuid: "779e66f2e58a4f6ab3e310b66dbade52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260809 07:55:57.064235 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.059771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.28.112.2:32773)
0809 07:55:57.059879 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:57.059883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:57.059906 (+ 23us) server_negotiation.cc:407] Connection header received
0809 07:55:57.060110 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:57.060113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:57.060186 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:57.060292 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:55:57.060609 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.061337 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.062557 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.062886 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.062992 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:57.063437 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:55:57.063834 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:55:57.063925 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:57.064030 (+ 105us) server_negotiation.cc:299] Negotiation successful
0809 07:55:57.064080 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 07:55:57.069927 18348 raft_consensus.cc:1261] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc [term 1 FOLLOWER]: Refusing update from remote peer 779e66f2e58a4f6ab3e310b66dbade52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:55:57.069926 17945 raft_consensus.cc:1261] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Refusing update from remote peer 779e66f2e58a4f6ab3e310b66dbade52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:55:57.070644 18458 consensus_queue.cc:1035] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:55:57.070834 18454 consensus_queue.cc:1035] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 07:55:57.075838 18421 tablet.cc:2369] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:55:57.078410 18463 mvcc.cc:205] Tried to move back new op lower bound from 7316529795351134208 to 7316529795219546112. Current Snapshot: MvccSnapshot[applied={T|T < 7316529795351134208}]
I20260809 07:55:57.081826 18464 mvcc.cc:205] Tried to move back new op lower bound from 7316529795351134208 to 7316529795219546112. Current Snapshot: MvccSnapshot[applied={T|T < 7316529795351134208}]
I20260809 07:55:57.086521 17714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:55:57.086684 17714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:55:57.088987 17714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d: Sending DeleteTablet for 3 replicas of tablet a16c64a9eb1c453c8277a3eff5b72896
I20260809 07:55:57.089700 18323 tablet_service.cc:1514] Processing DeleteTablet for tablet a16c64a9eb1c453c8277a3eff5b72896 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:55:57 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:58434
I20260809 07:55:57.089751 18108 tablet_service.cc:1514] Processing DeleteTablet for tablet a16c64a9eb1c453c8277a3eff5b72896 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:55:57 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:39680
I20260809 07:55:57.090307 18473 tablet_replica.cc:330] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc: stopping tablet replica
I20260809 07:55:57.090314 17913 tablet_service.cc:1514] Processing DeleteTablet for tablet a16c64a9eb1c453c8277a3eff5b72896 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:55:57 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:33004
I20260809 07:55:57.090664 18473 raft_consensus.cc:2229] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:55:57.090972 18473 raft_consensus.cc:2258] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:55:57.091060 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 17812
I20260809 07:55:57.118875 18474 tablet_replica.cc:330] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52: stopping tablet replica
I20260809 07:55:57.119355 18474 raft_consensus.cc:2229] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:55:57.119896 18474 raft_consensus.cc:2258] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:55:57.121554 18474 ts_tablet_manager.cc:1905] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:55:57.121559 18473 ts_tablet_manager.cc:1905] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:55:57.124491 18473 ts_tablet_manager.cc:1918] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:55:57.124508 18474 ts_tablet_manager.cc:1918] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:55:57.124622 18474 log.cc:1196] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/a16c64a9eb1c453c8277a3eff5b72896
I20260809 07:55:57.124697 18473 log.cc:1196] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/a16c64a9eb1c453c8277a3eff5b72896
I20260809 07:55:57.125032 18473 ts_tablet_manager.cc:1939] T a16c64a9eb1c453c8277a3eff5b72896 P 7113fd4151764215a0c7cada8f56ebcc: Deleting consensus metadata
I20260809 07:55:57.125144 18474 ts_tablet_manager.cc:1939] T a16c64a9eb1c453c8277a3eff5b72896 P 779e66f2e58a4f6ab3e310b66dbade52: Deleting consensus metadata
I20260809 07:55:57.126495 17695 catalog_manager.cc:4841] TS 779e66f2e58a4f6ab3e310b66dbade52 (127.28.112.2:32773): tablet a16c64a9eb1c453c8277a3eff5b72896 (table test-table [id=caf3c7f8ac404e4ebc56e32601df2b85]) successfully deleted
I20260809 07:55:57.126497 17699 catalog_manager.cc:4841] TS 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997): tablet a16c64a9eb1c453c8277a3eff5b72896 (table test-table [id=caf3c7f8ac404e4ebc56e32601df2b85]) successfully deleted
W20260809 07:55:57.127285 17695 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.112.1:35403 (error 108)
I20260809 07:55:57.127327 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 18019
I20260809 07:55:57.136083 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.135684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:35403 (local address 127.0.0.1:33006)
0809 07:55:57.135864 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:57.135971 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:35403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260809 07:55:57.136343 17695 catalog_manager.cc:4584] TS 1fbc44e8df044db58a770554ee292de6 (127.28.112.1:35403): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a16c64a9eb1c453c8277a3eff5b72896: Network error: Client connection negotiation failed: client connection to 127.28.112.1:35403: connect: Connection refused (error 111)
I20260809 07:55:57.141403 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 18231
I20260809 07:55:57.152540 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 17671
I20260809 07:55:57.161728 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:43185
--webserver_interface=127.28.112.62
--webserver_port=46003
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:43185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:55:57.271032 18477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:57.271278 18477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:57.271332 18477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:57.275219 18477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:55:57.275271 18477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:57.275297 18477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:57.275319 18477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:55:57.275347 18477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:55:57.280448 18477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:43185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:43185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=46003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:57.281486 18477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:57.282460 18477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:57.295328 18491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:57.295404 18477 server_base.cc:1034] running on GCE node
W20260809 07:55:57.295478 18494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:57.295327 18490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:57.296178 18477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:57.297276 18477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:57.298430 18477 hybrid_clock.cc:648] HybridClock initialized: now 1786262157298411 us; error 31 us; skew 500 ppm
Aug 09 07:55:57 dist-test-slave-vpvm krb5kdc[17657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262157, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:57.301033 18477 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:55:57.301939 18477 webserver.cc:460] Webserver started at http://127.28.112.62:46003/ using document root <none> and password file <none>
I20260809 07:55:57.302547 18477 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:57.302641 18477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:57.306464 18477 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260809 07:55:57.308485 18503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:57.309615 18477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:55:57.309743 18477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "778a7b6b0e6a40118c54ad5712e7771d"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "1baa5122156ccaff2315d975fe33a374"
server_key_iv: "ebd99b78a01fbf91592a3ca784b42cc7"
server_key_version: "encryptionkey@0"
I20260809 07:55:57.310367 18477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:57.325490 18477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:57.328243 18477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:57.328474 18477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:57.335767 18477 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:43185
I20260809 07:55:57.336055 18591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:43185 every 8 connection(s)
I20260809 07:55:57.336907 18477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:55:57.337668 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 18477
I20260809 07:55:57.338128 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:35403
--local_ip_for_outbound_sockets=127.28.112.1
--tserver_master_addrs=127.28.112.62:43185
--webserver_port=39571
--webserver_interface=127.28.112.1
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 07:55:57.341915 18592 sys_catalog.cc:263] Verifying existing consensus state
I20260809 07:55:57.343623 18592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d: Bootstrap starting.
I20260809 07:55:57.360018 18592 log.cc:824] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:57.368687 18592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 07:55:57.369024 18592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d: Bootstrap complete.
I20260809 07:55:57.372849 18592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a7b6b0e6a40118c54ad5712e7771d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43185 } }
I20260809 07:55:57.373302 18592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778a7b6b0e6a40118c54ad5712e7771d, State: Initialized, Role: FOLLOWER
I20260809 07:55:57.373906 18592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a7b6b0e6a40118c54ad5712e7771d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43185 } }
I20260809 07:55:57.374083 18592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:55:57.374178 18592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:55:57.374290 18592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 1 FOLLOWER]: Advancing to term 2
I20260809 07:55:57.376132 18592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a7b6b0e6a40118c54ad5712e7771d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43185 } }
I20260809 07:55:57.376456 18592 leader_election.cc:302] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 778a7b6b0e6a40118c54ad5712e7771d; no voters:
I20260809 07:55:57.376750 18592 leader_election.cc:288] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 07:55:57.376886 18597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 2 FOLLOWER]: Leader election won for term 2
I20260809 07:55:57.377257 18597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 2 LEADER]: Becoming Leader. State: Replica: 778a7b6b0e6a40118c54ad5712e7771d, State: Running, Role: LEADER
I20260809 07:55:57.377578 18597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a7b6b0e6a40118c54ad5712e7771d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43185 } }
I20260809 07:55:57.377909 18592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:55:57.378672 18597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 778a7b6b0e6a40118c54ad5712e7771d. Latest consensus state: current_term: 2 leader_uuid: "778a7b6b0e6a40118c54ad5712e7771d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a7b6b0e6a40118c54ad5712e7771d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43185 } } }
I20260809 07:55:57.378911 18597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [sys.catalog]: This master's current role is: LEADER
I20260809 07:55:57.379040 18598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "778a7b6b0e6a40118c54ad5712e7771d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a7b6b0e6a40118c54ad5712e7771d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43185 } } }
I20260809 07:55:57.379232 18598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [sys.catalog]: This master's current role is: LEADER
I20260809 07:55:57.379349 18612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:55:57.384349 18612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:55:57.385401 18612 catalog_manager.cc:1259] Loaded cluster ID: b05d8f3e7cde437daca2bc708fc91895
I20260809 07:55:57.385471 18612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:55:57.386986 18612 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260809 07:55:57.391456 18612 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260809 07:55:57.396728 18612 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:55:57.397477 18612 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d: Loaded TSK: 0
I20260809 07:55:57.398739 18612 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 07:55:57.457624 18595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:57.457859 18595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:57.457901 18595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:57.461908 18595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:57.461992 18595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:57.462086 18595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:55:57.467221 18595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:35403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=39571
--tserver_master_addrs=127.28.112.62:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:57.468127 18595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:57.469069 18595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:57.478283 18630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:57.478302 18633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:57.478546 18595 server_base.cc:1034] running on GCE node
W20260809 07:55:57.478701 18635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:57.479116 18595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:57.479774 18595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:57.480948 18595 hybrid_clock.cc:648] HybridClock initialized: now 1786262157480928 us; error 36 us; skew 500 ppm
Aug 09 07:55:57 dist-test-slave-vpvm krb5kdc[17657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262157, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:57.483783 18595 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:55:57.484686 18595 webserver.cc:460] Webserver started at http://127.28.112.1:39571/ using document root <none> and password file <none>
I20260809 07:55:57.485210 18595 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:57.485294 18595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:57.488956 18595 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:55:57.491076 18645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:57.492749 18595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 07:55:57.492857 18595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "1fbc44e8df044db58a770554ee292de6"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "a6b45360ad9baaeb79d15e2c8b0008f1"
server_key_iv: "3a7c6f36b4cedb2974a1e3f5a4063d97"
server_key_version: "encryptionkey@0"
I20260809 07:55:57.493367 18595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:57.510291 18595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:57.513319 18595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:57.513494 18595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:57.514118 18595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:55:57.515244 18652 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 07:55:57.519457 18595 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 07:55:57.519512 18595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260809 07:55:57.519559 18595 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 07:55:57.521143 18595 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 07:55:57.521212 18595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260809 07:55:57.521488 18652 tablet_bootstrap.cc:492] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6: Bootstrap starting.
I20260809 07:55:57.532867 18595 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:35403
I20260809 07:55:57.532877 18811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:35403 every 8 connection(s)
I20260809 07:55:57.534245 18595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:55:57.535647 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 18595
I20260809 07:55:57.536090 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:32773
--local_ip_for_outbound_sockets=127.28.112.2
--tserver_master_addrs=127.28.112.62:43185
--webserver_port=41211
--webserver_interface=127.28.112.2
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 07:55:57.542585 18652 log.cc:824] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6: Log is configured to *not* fsync() on all Append() calls
Aug 09 07:55:57 dist-test-slave-vpvm krb5kdc[17657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262157, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:55:57.551051 18652 tablet_bootstrap.cc:492] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 07:55:57.551441 18652 tablet_bootstrap.cc:492] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6: Bootstrap complete.
I20260809 07:55:57.552587 18652 ts_tablet_manager.cc:1397] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.008s
I20260809 07:55:57.554028 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.536709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:37805)
0809 07:55:57.537517 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:57.537571 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:57.538652 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:57.540117 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:57.540123 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:57.540462 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:57.540902 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:57.540916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.542831 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.542835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:57.543885 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:57.543899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.544953 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.544957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:57.545162 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:57.545864 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:57.545887 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:57.547857 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:57.550470 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:57.550476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:57.550489 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:57.550837 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:57.551225 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:57.551229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:57.551231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:57.551409 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:57.552147 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:57.552154 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:57.552441 (+ 287us) client_negotiation.cc:769] Sending connection context
0809 07:55:57.552640 (+ 199us) client_negotiation.cc:240] Negotiation successful
0809 07:55:57.553097 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":152,"threads_started":1}
I20260809 07:55:57.554026 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.536739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:37805 (local address 127.28.112.62:43185)
0809 07:55:57.537221 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:57.537241 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:57.538078 (+ 837us) server_negotiation.cc:407] Connection header received
0809 07:55:57.539195 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:57.539212 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:57.539655 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:57.540093 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:57.541080 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.542638 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.544051 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.544788 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.544965 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:57.548080 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:57.548087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:57.548105 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:57.548141 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:57.550341 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:57.550954 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:57.550961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:57.550966 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:57.551112 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:57.551527 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:57.551531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:57.551533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:57.552013 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:57.552153 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:57.552802 (+ 649us) server_negotiation.cc:299] Negotiation successful
0809 07:55:57.553121 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":114,"threads_started":1}
I20260809 07:55:57.555471 18652 raft_consensus.cc:348] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } }
I20260809 07:55:57.555874 18652 raft_consensus.cc:729] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fbc44e8df044db58a770554ee292de6, State: Initialized, Role: FOLLOWER
I20260809 07:55:57.556370 18652 consensus_queue.cc:260] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } }
I20260809 07:55:57.557109 18812 heartbeater.cc:347] Connected to a master server at 127.28.112.62:43185
I20260809 07:55:57.557340 18652 ts_tablet_manager.cc:1428] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 07:55:57.557348 18812 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:57.557915 18812 heartbeater.cc:511] Master 127.28.112.62:43185 requested a full tablet report, sending...
I20260809 07:55:57.560523 18537 ts_manager.cc:194] Registered new tserver with Master: 1fbc44e8df044db58a770554ee292de6 (127.28.112.1:35403)
I20260809 07:55:57.564049 18537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:37805
I20260809 07:55:57.566592 18812 heartbeater.cc:503] Master 127.28.112.62:43185 was elected leader, sending a full tablet report...
Aug 09 07:55:57 dist-test-slave-vpvm krb5kdc[17657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262157, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for kudu/127.28.112.1@KRBTEST.COM
I20260809 07:55:57.574467 18825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.563449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:35403 (local address 127.0.0.1:33018)
0809 07:55:57.564260 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:57.564303 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:57.564456 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:57.565043 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:57.565049 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:57.565459 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:57.565599 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:57.565607 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.566894 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.566897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:57.567806 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:57.567818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.568539 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.568543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:57.568651 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:57.569165 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:57.569178 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:57.570735 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:57.572762 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:57.572771 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:57.572775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:57.573004 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:57.573375 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:57.573378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:57.573379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:57.573468 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:57.573935 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:57.573940 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:57.574072 (+ 132us) client_negotiation.cc:769] Sending connection context
0809 07:55:57.574190 (+ 118us) client_negotiation.cc:240] Negotiation successful
0809 07:55:57.574338 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":114,"threads_started":1}
I20260809 07:55:57.574605 18827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.563523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33018 (local address 127.28.112.1:35403)
0809 07:55:57.563896 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:57.563907 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:57.564601 (+ 694us) server_negotiation.cc:407] Connection header received
0809 07:55:57.564676 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:57.564682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:57.564887 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:57.565034 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:57.565735 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.566777 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.567974 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.568414 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.568506 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:57.570883 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:57.570894 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:57.570907 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:57.570934 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:57.572651 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:57.573163 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:57.573169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:57.573171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:57.573261 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:57.573558 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:57.573561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:57.573562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:57.573834 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:57.573979 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:57.574278 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 07:55:57.574440 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":95,"threads_started":1}
I20260809 07:55:57.575598 18715 tablet_service.cc:1514] Processing DeleteTablet for tablet a16c64a9eb1c453c8277a3eff5b72896 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:55:57 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:33018
I20260809 07:55:57.576391 18828 tablet_replica.cc:330] stopping tablet replica
I20260809 07:55:57.576650 18828 raft_consensus.cc:2229] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:55:57.576860 18828 raft_consensus.cc:2258] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:55:57.577816 18828 ts_tablet_manager.cc:1905] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:55:57.581722 18828 ts_tablet_manager.cc:1918] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:55:57.581812 18828 log.cc:1196] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/wal/wals/a16c64a9eb1c453c8277a3eff5b72896
I20260809 07:55:57.582101 18828 ts_tablet_manager.cc:1939] T a16c64a9eb1c453c8277a3eff5b72896 P 1fbc44e8df044db58a770554ee292de6: Deleting consensus metadata
I20260809 07:55:57.583735 18507 catalog_manager.cc:4841] TS 1fbc44e8df044db58a770554ee292de6 (127.28.112.1:35403): tablet a16c64a9eb1c453c8277a3eff5b72896 (table test-table [id=caf3c7f8ac404e4ebc56e32601df2b85]) successfully deleted
W20260809 07:55:57.655556 18820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:57.655786 18820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:57.655843 18820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:57.659690 18820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:57.659740 18820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:57.659814 18820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:55:57.665131 18820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:32773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=41211
--tserver_master_addrs=127.28.112.62:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:57.666079 18820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:57.667013 18820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:57.676342 18838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:57.676389 18834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:57.676337 18835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:57.676601 18820 server_base.cc:1034] running on GCE node
I20260809 07:55:57.677016 18820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:57.677569 18820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:57.678717 18820 hybrid_clock.cc:648] HybridClock initialized: now 1786262157678699 us; error 29 us; skew 500 ppm
Aug 09 07:55:57 dist-test-slave-vpvm krb5kdc[17657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262157, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:57.681224 18820 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:55:57.682009 18820 webserver.cc:460] Webserver started at http://127.28.112.2:41211/ using document root <none> and password file <none>
I20260809 07:55:57.682502 18820 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:57.682571 18820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:57.686147 18820 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260809 07:55:57.687991 18853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:57.689071 18820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:55:57.689172 18820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "779e66f2e58a4f6ab3e310b66dbade52"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "3e8e3533041c0b25cc8dab5779a791ad"
server_key_iv: "39117c70ec5ec17281ea145b61e411e2"
server_key_version: "encryptionkey@0"
I20260809 07:55:57.689638 18820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:57.698696 18820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:57.701210 18820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:57.701342 18820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:57.701810 18820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:55:57.702597 18820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:55:57.702647 18820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:57.702684 18820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:55:57.702698 18820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:57.710925 18820 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:32773
I20260809 07:55:57.711014 19044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:32773 every 8 connection(s)
I20260809 07:55:57.711843 18820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:55:57.715296 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 18820
I20260809 07:55:57.715705 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:39997
--local_ip_for_outbound_sockets=127.28.112.3
--tserver_master_addrs=127.28.112.62:43185
--webserver_port=37685
--webserver_interface=127.28.112.3
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 07:55:57 dist-test-slave-vpvm krb5kdc[17657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262157, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:55:57.725376 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.713568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:43125 (local address 127.28.112.62:43185)
0809 07:55:57.713685 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:57.713688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:57.714579 (+ 891us) server_negotiation.cc:407] Connection header received
0809 07:55:57.716009 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:57.716012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:57.716074 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:57.716160 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:57.717257 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.717777 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.718914 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.719237 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.719337 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:57.721775 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:57.721780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:57.721783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:57.721815 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:57.723401 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:57.723888 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:57.723893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:57.723894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:57.723989 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:57.724300 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:57.724303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:57.724305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:57.724529 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:57.724636 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:57.725064 (+ 428us) server_negotiation.cc:299] Negotiation successful
0809 07:55:57.725209 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 07:55:57.726087 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.713977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:43125)
0809 07:55:57.714414 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:57.714490 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:57.715759 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:57.716339 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:57.716345 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:57.716677 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:57.717113 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:57.717125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.717897 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.717900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:57.718792 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:57.718803 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.719418 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.719422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:57.719572 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:57.720104 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:57.720122 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:57.721590 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:57.723525 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:57.723530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:57.723540 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:57.723788 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:57.724081 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:57.724083 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:57.724084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:57.724216 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:57.724623 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:57.724628 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:57.724838 (+ 210us) client_negotiation.cc:769] Sending connection context
0809 07:55:57.724995 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 07:55:57.725245 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":108,"threads_started":1}
I20260809 07:55:57.727209 19045 heartbeater.cc:347] Connected to a master server at 127.28.112.62:43185
I20260809 07:55:57.727442 19045 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:57.727923 19045 heartbeater.cc:511] Master 127.28.112.62:43185 requested a full tablet report, sending...
I20260809 07:55:57.728958 18537 ts_manager.cc:194] Registered new tserver with Master: 779e66f2e58a4f6ab3e310b66dbade52 (127.28.112.2:32773)
I20260809 07:55:57.729538 18537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:43125
W20260809 07:55:57.827554 19050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:57.827814 19050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:57.827883 19050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:57.831914 19050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:57.831982 19050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:57.832069 19050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:55:57.837113 19050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:39997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=37685
--tserver_master_addrs=127.28.112.62:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:57.838191 19050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:57.839190 19050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:57.848320 19063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:57.848356 19059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:57.848356 19060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:57.848558 19050 server_base.cc:1034] running on GCE node
I20260809 07:55:57.848954 19050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:57.849642 19050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:57.850811 19050 hybrid_clock.cc:648] HybridClock initialized: now 1786262157850778 us; error 43 us; skew 500 ppm
Aug 09 07:55:57 dist-test-slave-vpvm krb5kdc[17657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262157, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:57.853360 19050 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:55:57.854221 19050 webserver.cc:460] Webserver started at http://127.28.112.3:37685/ using document root <none> and password file <none>
I20260809 07:55:57.854720 19050 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:57.854795 19050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:57.858414 19050 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:55:57.860257 19074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:57.861382 19050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:55:57.861492 19050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "7113fd4151764215a0c7cada8f56ebcc"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "f1e26a4af2be627a4e02bce7713d2255"
server_key_iv: "698ebd02c689894bb1a128163f589a70"
server_key_version: "encryptionkey@0"
I20260809 07:55:57.862048 19050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:57.890812 19050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:57.893872 19050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:57.894065 19050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:57.894587 19050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:55:57.895352 19050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:55:57.895398 19050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:57.895449 19050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:55:57.895468 19050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:57.903790 19050 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:39997
I20260809 07:55:57.903846 19273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:39997 every 8 connection(s)
I20260809 07:55:57.904721 19050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:55:57.913283 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 19050
Aug 09 07:55:57 dist-test-slave-vpvm krb5kdc[17657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262157, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:55:57.918002 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.906418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:45803 (local address 127.28.112.62:43185)
0809 07:55:57.906537 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:57.906541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:57.907356 (+ 815us) server_negotiation.cc:407] Connection header received
0809 07:55:57.908618 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:57.908621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:57.908668 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:57.908739 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:57.909812 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.910378 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.911470 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.911702 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.911803 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:57.914288 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:57.914293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:57.914296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:57.914323 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:57.915999 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:57.916520 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:57.916523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:57.916524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:57.916607 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:57.916966 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:57.916969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:57.916970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:57.917145 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:57.917220 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:57.917730 (+ 510us) server_negotiation.cc:299] Negotiation successful
0809 07:55:57.917837 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 07:55:57.918828 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.906768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:45803)
0809 07:55:57.907215 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:57.907254 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:57.908426 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:57.908883 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:57.908889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:57.909226 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:55:57.909664 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:57.909676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.910484 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.910487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:57.911331 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:57.911340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.911827 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.911830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:57.911966 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:57.912445 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:55:57.912461 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:55:57.914129 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:55:57.916112 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:57.916117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:57.916128 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:57.916405 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:57.916697 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:55:57.916700 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:55:57.916702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:55:57.916870 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:55:57.917242 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:55:57.917249 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:55:57.917482 (+ 233us) client_negotiation.cc:769] Sending connection context
0809 07:55:57.917672 (+ 190us) client_negotiation.cc:240] Negotiation successful
0809 07:55:57.918028 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":123,"threads_started":1}
I20260809 07:55:57.920162 19276 heartbeater.cc:347] Connected to a master server at 127.28.112.62:43185
I20260809 07:55:57.920384 19276 heartbeater.cc:464] Registering TS with master...
I20260809 07:55:57.920855 19276 heartbeater.cc:511] Master 127.28.112.62:43185 requested a full tablet report, sending...
I20260809 07:55:57.921833 18537 ts_manager.cc:194] Registered new tserver with Master: 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997)
I20260809 07:55:57.922076 19279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.914062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.28.112.62:43185)
0809 07:55:57.914293 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:57.914298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:57.914512 (+ 214us) server_negotiation.cc:407] Connection header received
0809 07:55:57.914602 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:57.914606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:57.914654 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:57.914733 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:55:57.915521 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.916251 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.917294 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.917524 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.917598 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:57.918939 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:55:57.918943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:55:57.918945 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:55:57.918970 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:55:57.920429 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:57.920814 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:57.920818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:57.920820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:57.920900 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:55:57.921181 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:55:57.921185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:55:57.921187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:55:57.921350 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:55:57.921534 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:57.921700 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 07:55:57.921792 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
I20260809 07:55:57.922508 18537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:45803
I20260809 07:55:57.923728 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 07:55:57.927305 19279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.924310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57364 (local address 127.28.112.62:43185)
0809 07:55:57.924413 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:57.924417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:57.924522 (+ 105us) server_negotiation.cc:407] Connection header received
0809 07:55:57.924631 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:57.924633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:57.924710 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:57.924790 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:55:57.924921 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.925376 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.926238 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.926441 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.926511 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:57.926737 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:55:57.927045 (+ 308us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:55:57.927125 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:57.927162 (+ 37us) server_negotiation.cc:299] Negotiation successful
0809 07:55:57.927204 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 07:55:57.927892 18537 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:55:57.930990 18537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:55:57.938920 18715 tablet_service.cc:1467] Processing CreateTablet for tablet 1d76d931b9524c1ca4ad2f7b7a2f1891 (DEFAULT_TABLE table=test-table [id=3625e6e11cd2405584fc9d0895f56394]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:55:57.939538 18715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d76d931b9524c1ca4ad2f7b7a2f1891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:55:57.942339 18652 tablet_bootstrap.cc:492] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6: Bootstrap starting.
I20260809 07:55:57.942698 18825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.937285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:32773 (local address 127.0.0.1:39694)
0809 07:55:57.937399 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:57.937414 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:57.937487 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:57.938325 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:57.938328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:57.938344 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:55:57.938429 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:57.938435 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.940358 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.940362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:57.941162 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:57.941171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.942295 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.942299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:57.942402 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:57.942448 (+ 46us) client_negotiation.cc:769] Sending connection context
0809 07:55:57.942500 (+ 52us) client_negotiation.cc:240] Negotiation successful
0809 07:55:57.942575 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":1}
I20260809 07:55:57.942818 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.937631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:39997 (local address 127.0.0.1:58446)
0809 07:55:57.937847 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:57.937859 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:57.937941 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:57.938570 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:57.938573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:57.938590 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:55:57.938672 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:57.938677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.940020 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.940023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:57.941213 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:57.941225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.942475 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.942478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:57.942569 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:57.942588 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 07:55:57.942657 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 07:55:57.942714 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":29,"thread_start_us":95,"threads_started":1}
I20260809 07:55:57.942955 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.937892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.28.112.3:39997)
0809 07:55:57.938169 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:57.938182 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:57.938204 (+ 22us) server_negotiation.cc:407] Connection header received
0809 07:55:57.938271 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:57.938276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:57.938458 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:57.938590 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:55:57.938757 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.939884 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.941319 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.942157 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.942262 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:57.942403 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:57.942712 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 07:55:57.942819 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":111,"threads_started":1}
I20260809 07:55:57.943224 18652 tablet_bootstrap.cc:654] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6: Neither blocks nor log segments found. Creating new log.
I20260809 07:55:57.943854 19288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:57.937481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39694 (local address 127.28.112.2:32773)
0809 07:55:57.937842 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:57.937854 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:57.937878 (+ 24us) server_negotiation.cc:407] Connection header received
0809 07:55:57.937987 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:57.937993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:57.938193 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:57.938329 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:55:57.939235 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.940240 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.941274 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:57.942157 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:57.943434 (+ 1277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:57.943580 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:57.943628 (+ 48us) server_negotiation.cc:299] Negotiation successful
0809 07:55:57.943712 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":130,"threads_started":1}
I20260809 07:55:57.944604 18652 tablet_bootstrap.cc:492] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6: No bootstrap required, opened a new log
I20260809 07:55:57.944700 18652 ts_tablet_manager.cc:1397] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 07:55:57.945150 18652 raft_consensus.cc:348] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } }
I20260809 07:55:57.945248 18652 raft_consensus.cc:374] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:55:57.945284 18652 raft_consensus.cc:729] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fbc44e8df044db58a770554ee292de6, State: Initialized, Role: FOLLOWER
I20260809 07:55:57.945407 18652 consensus_queue.cc:260] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } }
I20260809 07:55:57.945760 18652 ts_tablet_manager.cc:1428] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 07:55:57.946274 19157 tablet_service.cc:1467] Processing CreateTablet for tablet 1d76d931b9524c1ca4ad2f7b7a2f1891 (DEFAULT_TABLE table=test-table [id=3625e6e11cd2405584fc9d0895f56394]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:55:57.946537 18931 tablet_service.cc:1467] Processing CreateTablet for tablet 1d76d931b9524c1ca4ad2f7b7a2f1891 (DEFAULT_TABLE table=test-table [id=3625e6e11cd2405584fc9d0895f56394]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:55:57.947382 18931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d76d931b9524c1ca4ad2f7b7a2f1891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:55:57.947394 19157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d76d931b9524c1ca4ad2f7b7a2f1891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:55:57.955333 19292 tablet_bootstrap.cc:492] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc: Bootstrap starting.
I20260809 07:55:57.955821 19293 tablet_bootstrap.cc:492] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52: Bootstrap starting.
I20260809 07:55:57.957715 19292 tablet_bootstrap.cc:654] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc: Neither blocks nor log segments found. Creating new log.
I20260809 07:55:57.958478 19293 tablet_bootstrap.cc:654] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52: Neither blocks nor log segments found. Creating new log.
I20260809 07:55:57.959115 19292 log.cc:824] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:57.959232 19293 log.cc:824] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:57.960826 19292 tablet_bootstrap.cc:492] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc: No bootstrap required, opened a new log
I20260809 07:55:57.960853 19293 tablet_bootstrap.cc:492] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52: No bootstrap required, opened a new log
I20260809 07:55:57.961011 19292 ts_tablet_manager.cc:1397] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260809 07:55:57.961027 19293 ts_tablet_manager.cc:1397] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260809 07:55:57.964015 19293 raft_consensus.cc:348] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } }
I20260809 07:55:57.964010 19292 raft_consensus.cc:348] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } }
I20260809 07:55:57.964207 19293 raft_consensus.cc:374] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:55:57.964210 19292 raft_consensus.cc:374] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:55:57.964246 19293 raft_consensus.cc:729] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 779e66f2e58a4f6ab3e310b66dbade52, State: Initialized, Role: FOLLOWER
I20260809 07:55:57.964258 19292 raft_consensus.cc:729] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7113fd4151764215a0c7cada8f56ebcc, State: Initialized, Role: FOLLOWER
I20260809 07:55:57.964643 19292 consensus_queue.cc:260] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } }
I20260809 07:55:57.964641 19293 consensus_queue.cc:260] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } }
I20260809 07:55:57.965255 19045 heartbeater.cc:503] Master 127.28.112.62:43185 was elected leader, sending a full tablet report...
I20260809 07:55:57.965281 19276 heartbeater.cc:503] Master 127.28.112.62:43185 was elected leader, sending a full tablet report...
I20260809 07:55:57.965619 19292 ts_tablet_manager.cc:1428] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260809 07:55:57.965654 19293 ts_tablet_manager.cc:1428] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
W20260809 07:55:58.035524 18813 tablet.cc:2369] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:55:58.072000 18823 raft_consensus.cc:482] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:55:58.072203 18823 raft_consensus.cc:504] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } }
I20260809 07:55:58.073294 18823 leader_election.cc:288] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 779e66f2e58a4f6ab3e310b66dbade52 (127.28.112.2:32773), 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997)
I20260809 07:55:58.077136 19288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:58.073603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:47849 (local address 127.28.112.2:32773)
0809 07:55:58.073733 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:58.073738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:58.074089 (+ 351us) server_negotiation.cc:407] Connection header received
0809 07:55:58.074202 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:58.074205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:58.074254 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:58.074316 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:55:58.074568 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:58.075051 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:58.076206 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:58.076594 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:58.076698 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:58.076752 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:58.076952 (+ 200us) server_negotiation.cc:299] Negotiation successful
0809 07:55:58.077017 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 07:55:58.077168 19302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:58.073522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:32773 (local address 127.28.112.1:47849)
0809 07:55:58.073991 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:58.074004 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:58.074118 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:58.074330 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:58.074333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:58.074349 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:55:58.074471 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:58.074476 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:58.075146 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:58.075150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:58.076094 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:58.076104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:58.076698 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:58.076702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:58.076792 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:58.076857 (+ 65us) client_negotiation.cc:769] Sending connection context
0809 07:55:58.076919 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 07:55:58.077005 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":224,"thread_start_us":70,"threads_started":1}
I20260809 07:55:58.077163 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:58.073657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:33411 (local address 127.28.112.3:39997)
0809 07:55:58.073815 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:58.073819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:58.073931 (+ 112us) server_negotiation.cc:407] Connection header received
0809 07:55:58.074138 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:58.074141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:58.074183 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:58.074246 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:55:58.074503 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:58.075004 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:58.076221 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:58.076609 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:58.076739 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:58.076790 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:58.076972 (+ 182us) server_negotiation.cc:299] Negotiation successful
0809 07:55:58.077039 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 07:55:58.077190 19298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:58.073522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:39997 (local address 127.28.112.1:33411)
0809 07:55:58.073835 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:58.073850 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:58.073998 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:58.074267 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:55:58.074270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:55:58.074288 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:55:58.074406 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:58.074410 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:58.075105 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:58.075108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:58.076098 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:55:58.076109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:55:58.076739 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:55:58.076744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:55:58.076866 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:58.076884 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 07:55:58.076947 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 07:55:58.077039 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":103,"thread_start_us":93,"threads_started":1}
I20260809 07:55:58.077852 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d76d931b9524c1ca4ad2f7b7a2f1891" candidate_uuid: "1fbc44e8df044db58a770554ee292de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "779e66f2e58a4f6ab3e310b66dbade52" is_pre_election: true
I20260809 07:55:58.077847 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d76d931b9524c1ca4ad2f7b7a2f1891" candidate_uuid: "1fbc44e8df044db58a770554ee292de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7113fd4151764215a0c7cada8f56ebcc" is_pre_election: true
I20260809 07:55:58.078135 18964 raft_consensus.cc:2454] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fbc44e8df044db58a770554ee292de6 in term 0.
I20260809 07:55:58.078159 19198 raft_consensus.cc:2454] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fbc44e8df044db58a770554ee292de6 in term 0.
I20260809 07:55:58.078585 18646 leader_election.cc:302] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fbc44e8df044db58a770554ee292de6, 779e66f2e58a4f6ab3e310b66dbade52; no voters:
I20260809 07:55:58.078820 18823 raft_consensus.cc:2781] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:55:58.078907 18823 raft_consensus.cc:482] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:55:58.078958 18823 raft_consensus.cc:3037] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:55:58.079957 18823 raft_consensus.cc:504] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } }
I20260809 07:55:58.080266 18823 leader_election.cc:288] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 1 election: Requested vote from peers 779e66f2e58a4f6ab3e310b66dbade52 (127.28.112.2:32773), 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997)
I20260809 07:55:58.080605 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d76d931b9524c1ca4ad2f7b7a2f1891" candidate_uuid: "1fbc44e8df044db58a770554ee292de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "779e66f2e58a4f6ab3e310b66dbade52"
I20260809 07:55:58.080605 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d76d931b9524c1ca4ad2f7b7a2f1891" candidate_uuid: "1fbc44e8df044db58a770554ee292de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7113fd4151764215a0c7cada8f56ebcc"
I20260809 07:55:58.080724 18964 raft_consensus.cc:3037] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:55:58.080724 19198 raft_consensus.cc:3037] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:55:58.081986 19198 raft_consensus.cc:2454] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fbc44e8df044db58a770554ee292de6 in term 1.
I20260809 07:55:58.081980 18964 raft_consensus.cc:2454] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fbc44e8df044db58a770554ee292de6 in term 1.
I20260809 07:55:58.082257 18648 leader_election.cc:302] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fbc44e8df044db58a770554ee292de6, 7113fd4151764215a0c7cada8f56ebcc; no voters:
I20260809 07:55:58.082412 18823 raft_consensus.cc:2781] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:55:58.082590 18823 raft_consensus.cc:686] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 LEADER]: Becoming Leader. State: Replica: 1fbc44e8df044db58a770554ee292de6, State: Running, Role: LEADER
I20260809 07:55:58.082840 18823 consensus_queue.cc:237] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } }
I20260809 07:55:58.085927 18537 catalog_manager.cc:5495] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fbc44e8df044db58a770554ee292de6 (127.28.112.1). New cstate: current_term: 1 leader_uuid: "1fbc44e8df044db58a770554ee292de6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } health_report { overall_health: HEALTHY } } }
I20260809 07:55:58.133143 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:58.129600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33028 (local address 127.28.112.1:35403)
0809 07:55:58.129887 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0809 07:55:58.129894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 07:55:58.129905 (+ 11us) server_negotiation.cc:407] Connection header received
0809 07:55:58.130009 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:55:58.130012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:55:58.130072 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:55:58.130183 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:55:58.130418 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:58.130934 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:58.131736 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:55:58.131964 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:55:58.132051 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:55:58.132320 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:55:58.132749 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:55:58.132854 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 07:55:58.132941 (+ 87us) server_negotiation.cc:299] Negotiation successful
0809 07:55:58.132995 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":107,"threads_started":1}
I20260809 07:55:58.136942 18964 raft_consensus.cc:1261] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52 [term 1 FOLLOWER]: Refusing update from remote peer 1fbc44e8df044db58a770554ee292de6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:55:58.136942 19198 raft_consensus.cc:1261] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc [term 1 FOLLOWER]: Refusing update from remote peer 1fbc44e8df044db58a770554ee292de6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:55:58.137684 19303 consensus_queue.cc:1035] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:55:58.137856 18823 consensus_queue.cc:1035] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:55:58.146684 19311 mvcc.cc:205] Tried to move back new op lower bound from 7316529799722872832 to 7316529799509856256. Current Snapshot: MvccSnapshot[applied={T|T < 7316529799722872832}]
I20260809 07:55:58.153256 18535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:55:58.153419 18535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:55:58.155802 18535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d: Sending DeleteTablet for 3 replicas of tablet 1d76d931b9524c1ca4ad2f7b7a2f1891
I20260809 07:55:58.156478 18931 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d76d931b9524c1ca4ad2f7b7a2f1891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:55:58 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:39694
I20260809 07:55:58.156549 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 18595
I20260809 07:55:58.156505 19157 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d76d931b9524c1ca4ad2f7b7a2f1891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:55:58 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:58446
I20260809 07:55:58.164371 19326 tablet_replica.cc:330] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc: stopping tablet replica
I20260809 07:55:58.164683 19326 raft_consensus.cc:2229] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:55:58.164992 19326 raft_consensus.cc:2258] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:55:58.165011 19325 tablet_replica.cc:330] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52: stopping tablet replica
I20260809 07:55:58.165287 19325 raft_consensus.cc:2229] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:55:58.165586 19325 raft_consensus.cc:2258] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:55:58.166474 19326 ts_tablet_manager.cc:1905] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:55:58.166955 19325 ts_tablet_manager.cc:1905] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:55:58.169252 19326 ts_tablet_manager.cc:1918] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:55:58.169373 19326 log.cc:1196] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/1d76d931b9524c1ca4ad2f7b7a2f1891
I20260809 07:55:58.169534 19325 ts_tablet_manager.cc:1918] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:55:58.169641 19325 log.cc:1196] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/1d76d931b9524c1ca4ad2f7b7a2f1891
I20260809 07:55:58.169752 19326 ts_tablet_manager.cc:1939] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 7113fd4151764215a0c7cada8f56ebcc: Deleting consensus metadata
I20260809 07:55:58.170004 19325 ts_tablet_manager.cc:1939] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 779e66f2e58a4f6ab3e310b66dbade52: Deleting consensus metadata
W20260809 07:55:58.170388 18507 connection.cc:535] client connection to 127.28.112.1:35403 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 07:55:58.170523 18507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 07:55:58.171044 18511 catalog_manager.cc:4841] TS 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997): tablet 1d76d931b9524c1ca4ad2f7b7a2f1891 (table test-table [id=3625e6e11cd2405584fc9d0895f56394]) successfully deleted
I20260809 07:55:58.171044 18507 catalog_manager.cc:4841] TS 779e66f2e58a4f6ab3e310b66dbade52 (127.28.112.2:32773): tablet 1d76d931b9524c1ca4ad2f7b7a2f1891 (table test-table [id=3625e6e11cd2405584fc9d0895f56394]) successfully deleted
I20260809 07:55:58.171484 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 18820
I20260809 07:55:58.171733 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:58.171469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:35403 (local address 127.0.0.1:33036)
0809 07:55:58.171593 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:58.171659 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:35403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 07:55:58.173282 18507 catalog_manager.cc:4584] TS 1fbc44e8df044db58a770554ee292de6 (127.28.112.1:35403): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d76d931b9524c1ca4ad2f7b7a2f1891: Network error: Client connection negotiation failed: client connection to 127.28.112.1:35403: connect: Connection refused (error 111)
I20260809 07:55:58.182963 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 19050
I20260809 07:55:58.194994 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 18477
I20260809 07:55:58.207324 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:43185
--webserver_interface=127.28.112.62
--webserver_port=46003
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:43185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:55:58.317369 19328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:58.317610 19328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:58.317669 19328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:58.321569 19328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:55:58.321622 19328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:58.321650 19328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:58.321671 19328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:55:58.321700 19328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:55:58.326789 19328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:43185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:43185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=46003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:58.327720 19328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:58.328667 19328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:58.336932 19334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:58.336941 19339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:58.336956 19336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:58.337066 19328 server_base.cc:1034] running on GCE node
I20260809 07:55:58.337707 19328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:58.338798 19328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:58.339964 19328 hybrid_clock.cc:648] HybridClock initialized: now 1786262158339946 us; error 29 us; skew 500 ppm
Aug 09 07:55:58 dist-test-slave-vpvm krb5kdc[17657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262158, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:58.342571 19328 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:55:58.343467 19328 webserver.cc:460] Webserver started at http://127.28.112.62:46003/ using document root <none> and password file <none>
I20260809 07:55:58.343983 19328 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:58.344066 19328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:58.347738 19328 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:55:58.349786 19346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:58.350957 19328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:55:58.351084 19328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "778a7b6b0e6a40118c54ad5712e7771d"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "1baa5122156ccaff2315d975fe33a374"
server_key_iv: "ebd99b78a01fbf91592a3ca784b42cc7"
server_key_version: "encryptionkey@0"
I20260809 07:55:58.351604 19328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:58.378643 19328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:58.381126 19328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:58.381285 19328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:58.388024 19328 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:43185
I20260809 07:55:58.388274 19425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:43185 every 8 connection(s)
I20260809 07:55:58.389042 19328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:55:58.393158 19426 sys_catalog.cc:263] Verifying existing consensus state
I20260809 07:55:58.393462 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 19328
I20260809 07:55:58.393862 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:35403
--local_ip_for_outbound_sockets=127.28.112.1
--tserver_master_addrs=127.28.112.62:43185
--webserver_port=39571
--webserver_interface=127.28.112.1
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 07:55:58.394873 19426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d: Bootstrap starting.
I20260809 07:55:58.411275 19426 log.cc:824] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:58.422626 19426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 07:55:58.422971 19426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d: Bootstrap complete.
I20260809 07:55:58.426786 19426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a7b6b0e6a40118c54ad5712e7771d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43185 } }
I20260809 07:55:58.427168 19426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778a7b6b0e6a40118c54ad5712e7771d, State: Initialized, Role: FOLLOWER
I20260809 07:55:58.427619 19426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a7b6b0e6a40118c54ad5712e7771d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43185 } }
I20260809 07:55:58.427742 19426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:55:58.427803 19426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:55:58.427891 19426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 2 FOLLOWER]: Advancing to term 3
I20260809 07:55:58.429682 19426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a7b6b0e6a40118c54ad5712e7771d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43185 } }
I20260809 07:55:58.430019 19426 leader_election.cc:302] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 778a7b6b0e6a40118c54ad5712e7771d; no voters:
I20260809 07:55:58.430326 19426 leader_election.cc:288] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 07:55:58.430418 19431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 07:55:58.430588 19431 raft_consensus.cc:686] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [term 3 LEADER]: Becoming Leader. State: Replica: 778a7b6b0e6a40118c54ad5712e7771d, State: Running, Role: LEADER
I20260809 07:55:58.430877 19431 consensus_queue.cc:237] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a7b6b0e6a40118c54ad5712e7771d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43185 } }
I20260809 07:55:58.431527 19426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:55:58.431803 19431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 778a7b6b0e6a40118c54ad5712e7771d. Latest consensus state: current_term: 3 leader_uuid: "778a7b6b0e6a40118c54ad5712e7771d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a7b6b0e6a40118c54ad5712e7771d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43185 } } }
I20260809 07:55:58.432009 19431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [sys.catalog]: This master's current role is: LEADER
I20260809 07:55:58.432624 19432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "778a7b6b0e6a40118c54ad5712e7771d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778a7b6b0e6a40118c54ad5712e7771d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 43185 } } }
I20260809 07:55:58.432777 19432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778a7b6b0e6a40118c54ad5712e7771d [sys.catalog]: This master's current role is: LEADER
I20260809 07:55:58.433324 19438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:55:58.438223 19438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:55:58.438902 19438 catalog_manager.cc:1259] Loaded cluster ID: b05d8f3e7cde437daca2bc708fc91895
I20260809 07:55:58.438964 19438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260809 07:55:58.440088 19438 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff8377039ad google::LogMessage::Fail() at ??:0
@ 0x7ff83770688a google::LogMessage::SendToLog() at ??:0
@ 0x7ff8377034de google::LogMessage::Flush() at ??:0
@ 0x7ff837704a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff842046fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7ff842047d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff842044579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7ff842088640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff8399553d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff839955c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7ff8399573f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff839946512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff83af8a6db start_thread at ??:0
@ 0x7ff83620161f clone at ??:0
W20260809 07:55:58.524955 19429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:58.525179 19429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:58.525219 19429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:58.529129 19429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:58.529177 19429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:58.529234 19429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:55:58.534227 19429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:35403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=39571
--tserver_master_addrs=127.28.112.62:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:58.535143 19429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:58.536075 19429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:58.545059 19487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:58.545069 19483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:58.545069 19490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:58.545751 19429 server_base.cc:1034] running on GCE node
I20260809 07:55:58.546159 19429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:58.546730 19429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:58.547906 19429 hybrid_clock.cc:648] HybridClock initialized: now 1786262158547872 us; error 50 us; skew 500 ppm
Aug 09 07:55:58 dist-test-slave-vpvm krb5kdc[17657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262158, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:58.550776 19429 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:55:58.551620 19429 webserver.cc:460] Webserver started at http://127.28.112.1:39571/ using document root <none> and password file <none>
I20260809 07:55:58.552153 19429 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:58.552223 19429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:58.556107 19429 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 07:55:58.558295 19499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:58.559612 19429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 07:55:58.559733 19429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "1fbc44e8df044db58a770554ee292de6"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "a6b45360ad9baaeb79d15e2c8b0008f1"
server_key_iv: "3a7c6f36b4cedb2974a1e3f5a4063d97"
server_key_version: "encryptionkey@0"
I20260809 07:55:58.560261 19429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:58.569832 19429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:58.572803 19429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:58.572971 19429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:58.573519 19429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:55:58.574616 19509 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 07:55:58.580054 19429 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 07:55:58.580114 19429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260809 07:55:58.580175 19429 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 07:55:58.581743 19429 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 07:55:58.581786 19429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260809 07:55:58.582165 19509 tablet_bootstrap.cc:492] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6: Bootstrap starting.
I20260809 07:55:58.591254 19429 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:35403
I20260809 07:55:58.591475 19677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:35403 every 8 connection(s)
I20260809 07:55:58.592202 19429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:55:58.601258 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 19429
I20260809 07:55:58.601686 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:32773
--local_ip_for_outbound_sockets=127.28.112.2
--tserver_master_addrs=127.28.112.62:43185
--webserver_port=41211
--webserver_interface=127.28.112.2
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 07:55:58.604081 19509 log.cc:824] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6: Log is configured to *not* fsync() on all Append() calls
I20260809 07:55:58.613142 19509 tablet_bootstrap.cc:492] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 07:55:58.613579 19509 tablet_bootstrap.cc:492] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6: Bootstrap complete.
I20260809 07:55:58.614871 19509 ts_tablet_manager.cc:1397] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6: Time spent bootstrapping tablet: real 0.033s user 0.017s sys 0.015s
I20260809 07:55:58.618139 19509 raft_consensus.cc:348] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } }
I20260809 07:55:58.618673 19509 raft_consensus.cc:729] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fbc44e8df044db58a770554ee292de6, State: Initialized, Role: FOLLOWER
I20260809 07:55:58.619202 19509 consensus_queue.cc:260] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } }
I20260809 07:55:58.620265 19509 ts_tablet_manager.cc:1428] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260809 07:55:58.669479 19682 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:55:58.593970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:40077)
0809 07:55:58.594450 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:58.594501 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 07:55:58.595267 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:55:58.668725 (+ 73458us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":116,"threads_started":1}
W20260809 07:55:58.669940 19678 heartbeater.cc:644] Failed to heartbeat to 127.28.112.62:43185 (0 consecutive failures): Network error: Failed to ping master at 127.28.112.62:43185: Client connection negotiation failed: client connection to 127.28.112.62:43185: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 07:55:58.670719 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:58.670445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:49121)
0809 07:55:58.670569 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:58.670631 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260809 07:55:58.671355 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:58.671160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:49835)
0809 07:55:58.671264 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:58.671307 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 07:55:58.671523 19678 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 07:55:58.714257 19683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:58.714486 19683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:58.714532 19683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:58.718439 19683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:58.718489 19683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:58.718557 19683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:55:58.723536 19683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:32773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=41211
--tserver_master_addrs=127.28.112.62:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:58.724503 19683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:58.725462 19683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:58.734385 19696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:58.734427 19683 server_base.cc:1034] running on GCE node
W20260809 07:55:58.734395 19697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:58.734441 19699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:58.734988 19683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:58.735558 19683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:58.736729 19683 hybrid_clock.cc:648] HybridClock initialized: now 1786262158736710 us; error 33 us; skew 500 ppm
Aug 09 07:55:58 dist-test-slave-vpvm krb5kdc[17657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262158, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:58.739300 19683 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:55:58.740077 19683 webserver.cc:460] Webserver started at http://127.28.112.2:41211/ using document root <none> and password file <none>
I20260809 07:55:58.740579 19683 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:58.740643 19683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:58.744426 19683 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:55:58.746331 19707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:58.747476 19683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:55:58.747587 19683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "779e66f2e58a4f6ab3e310b66dbade52"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "3e8e3533041c0b25cc8dab5779a791ad"
server_key_iv: "39117c70ec5ec17281ea145b61e411e2"
server_key_version: "encryptionkey@0"
I20260809 07:55:58.748095 19683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:58.760627 19683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:58.763026 19683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:58.763192 19683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:58.763710 19683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:55:58.764461 19683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:55:58.764508 19683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:58.764557 19683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:55:58.764581 19683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:58.772768 19683 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:32773
I20260809 07:55:58.772790 19887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:32773 every 8 connection(s)
I20260809 07:55:58.773653 19683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:55:58.777107 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:58.775563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:40857)
0809 07:55:58.776012 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:58.776430 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":109,"threads_started":1}
W20260809 07:55:58.777537 19889 heartbeater.cc:644] Failed to heartbeat to 127.28.112.62:43185 (0 consecutive failures): Network error: Failed to ping master at 127.28.112.62:43185: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
I20260809 07:55:58.778060 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:58.777851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:58555)
0809 07:55:58.777963 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:58.778007 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260809 07:55:58.778393 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 19683
I20260809 07:55:58.778627 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:58.778439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:38111)
0809 07:55:58.778530 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:58.778579 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260809 07:55:58.778813 19889 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 07:55:58.778841 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:39997
--local_ip_for_outbound_sockets=127.28.112.3
--tserver_master_addrs=127.28.112.62:43185
--webserver_port=37685
--webserver_interface=127.28.112.3
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:55:58.887831 19894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:55:58.888072 19894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:55:58.888140 19894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:55:58.892132 19894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:55:58.892192 19894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:55:58.892277 19894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:55:58.897367 19894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:39997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=37685
--tserver_master_addrs=127.28.112.62:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:55:58.898373 19894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:55:58.899343 19894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:55:58.908463 19902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:55:58.908530 19904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:58.908663 19894 server_base.cc:1034] running on GCE node
W20260809 07:55:58.908531 19907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:55:58.909123 19894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:55:58.909703 19894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:55:58.910884 19894 hybrid_clock.cc:648] HybridClock initialized: now 1786262158910852 us; error 58 us; skew 500 ppm
Aug 09 07:55:58 dist-test-slave-vpvm krb5kdc[17657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262158, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:55:58.913628 19894 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:55:58.914484 19894 webserver.cc:460] Webserver started at http://127.28.112.3:37685/ using document root <none> and password file <none>
I20260809 07:55:58.914999 19894 fs_manager.cc:359] Metadata directory not provided
I20260809 07:55:58.915062 19894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:55:58.918570 19894 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:55:58.920421 19919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:58.921487 19894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:55:58.921589 19894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "7113fd4151764215a0c7cada8f56ebcc"
format_stamp: "Formatted at 2026-08-09 07:55:56 on dist-test-slave-vpvm"
server_key: "f1e26a4af2be627a4e02bce7713d2255"
server_key_iv: "698ebd02c689894bb1a128163f589a70"
server_key_version: "encryptionkey@0"
I20260809 07:55:58.922091 19894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:55:58.939548 19894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:55:58.942859 19894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:55:58.943032 19894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:55:58.943600 19894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:55:58.944389 19894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:55:58.944434 19894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:58.944487 19894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:55:58.944516 19894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:55:58.953089 19894 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:39997
I20260809 07:55:58.953116 20078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:39997 every 8 connection(s)
I20260809 07:55:58.954039 19894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:55:58.955344 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 19894
I20260809 07:55:58.957612 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:58.956081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:49149)
0809 07:55:58.956543 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:58.956943 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":102,"threads_started":1}
W20260809 07:55:58.958077 20079 heartbeater.cc:644] Failed to heartbeat to 127.28.112.62:43185 (0 consecutive failures): Network error: Failed to ping master at 127.28.112.62:43185: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
I20260809 07:55:58.958591 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:58.958388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:36645)
0809 07:55:58.958490 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:58.958536 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 07:55:58.959158 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:58.958968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:53097)
0809 07:55:58.959063 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:58.959106 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260809 07:55:58.959303 20079 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 07:55:59.672447 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:59.672017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:45977)
0809 07:55:59.672268 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:59.672323 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20260809 07:55:59.779627 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:59.779275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:41379)
0809 07:55:59.779485 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:59.779534 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":66,"threads_started":1}
I20260809 07:55:59.960191 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:55:59.959791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:34815)
0809 07:55:59.960036 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0809 07:55:59.960086 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":105,"threads_started":1}
I20260809 07:56:00.271706 20088 raft_consensus.cc:482] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:56:00.271905 20088 raft_consensus.cc:504] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } }
I20260809 07:56:00.273008 20088 leader_election.cc:288] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 779e66f2e58a4f6ab3e310b66dbade52 (127.28.112.2:32773), 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997)
Aug 09 07:56:00 dist-test-slave-vpvm krb5kdc[17657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262158, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.2@KRBTEST.COM
Aug 09 07:56:00 dist-test-slave-vpvm krb5kdc[17657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262158, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.3@KRBTEST.COM
I20260809 07:56:00.286854 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:00.273215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:39997 (local address 127.28.112.1:50443)
0809 07:56:00.273536 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:00.273550 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:00.273676 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:00.275539 (+ 1863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:00.275544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:00.275910 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:56:00.276369 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:00.276385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:00.277785 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:00.277787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:00.278810 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:00.278820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:00.279484 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:00.279487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:00.279670 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:00.280322 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:56:00.280333 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:56:00.282006 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:56:00.284009 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:00.284016 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:00.284030 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:00.284483 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:00.284850 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:00.284853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:00.284855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:00.285057 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:00.285767 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:56:00.285774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:56:00.286116 (+ 342us) client_negotiation.cc:769] Sending connection context
0809 07:56:00.286335 (+ 219us) client_negotiation.cc:240] Negotiation successful
0809 07:56:00.286637 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":213,"spinlock_wait_cycles":35200,"thread_start_us":71,"threads_started":1}
I20260809 07:56:00.286921 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:00.273374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:57689 (local address 127.28.112.2:32773)
0809 07:56:00.273763 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:00.273785 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:00.273974 (+ 189us) server_negotiation.cc:407] Connection header received
0809 07:56:00.274759 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:00.274765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:00.275138 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:00.275474 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:56:00.276533 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:00.277642 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:00.278918 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:00.279367 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:00.279522 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:00.281909 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:56:00.281914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:56:00.281926 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:56:00.281974 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:56:00.283879 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:00.284596 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:00.284602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:00.284608 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:00.284754 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:00.285146 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:00.285149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:00.285150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:00.285660 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:56:00.285785 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:00.286459 (+ 674us) server_negotiation.cc:299] Negotiation successful
0809 07:56:00.286742 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":100,"threads_started":1}
I20260809 07:56:00.286852 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:00.273215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:32773 (local address 127.28.112.1:57689)
0809 07:56:00.273861 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:00.273873 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:00.273979 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:00.275958 (+ 1979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:00.275961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:00.276127 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:56:00.276366 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:00.276381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:00.277786 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:00.277789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:00.278807 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:00.278818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:00.279487 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:00.279491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:00.279664 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:00.280215 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:56:00.280236 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:56:00.281697 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:56:00.284009 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:00.284015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:00.284030 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:00.284483 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:00.284850 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:00.284853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:00.284855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:00.285057 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:00.285767 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:56:00.285774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:56:00.286116 (+ 342us) client_negotiation.cc:769] Sending connection context
0809 07:56:00.286335 (+ 219us) client_negotiation.cc:240] Negotiation successful
0809 07:56:00.286698 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":635,"spinlock_wait_cycles":38656,"thread_start_us":74,"threads_started":1}
I20260809 07:56:00.286926 20090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:00.273380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:50443 (local address 127.28.112.3:39997)
0809 07:56:00.273645 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:00.273675 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:00.273722 (+ 47us) server_negotiation.cc:407] Connection header received
0809 07:56:00.274665 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:00.274676 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:00.275139 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:00.275474 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:56:00.276568 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:00.277646 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:00.278926 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:00.279368 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:00.279523 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:00.282217 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:56:00.282224 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:56:00.282240 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:56:00.282275 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:56:00.283911 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:00.284596 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:00.284602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:00.284608 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:00.284754 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:00.285150 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:00.285153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:00.285154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:00.285649 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:56:00.285785 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:00.286459 (+ 674us) server_negotiation.cc:299] Negotiation successful
0809 07:56:00.286742 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
W20260809 07:56:00.289645 19500 leader_election.cc:341] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 779e66f2e58a4f6ab3e310b66dbade52 (127.28.112.2:32773): Not found: Tablet not found: 1d76d931b9524c1ca4ad2f7b7a2f1891
W20260809 07:56:00.289847 19504 leader_election.cc:341] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997): Not found: Tablet not found: 1d76d931b9524c1ca4ad2f7b7a2f1891
I20260809 07:56:00.289930 19504 leader_election.cc:302] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fbc44e8df044db58a770554ee292de6; no voters: 7113fd4151764215a0c7cada8f56ebcc, 779e66f2e58a4f6ab3e310b66dbade52
I20260809 07:56:00.290211 20088 raft_consensus.cc:2726] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 07:56:00.673398 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:00.673098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:45295)
0809 07:56:00.673238 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:00.673295 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 07:56:00.780592 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:00.780233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:42213)
0809 07:56:00.780447 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:00.780497 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":70,"threads_started":1}
I20260809 07:56:00.961304 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:00.960890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:42303)
0809 07:56:00.961152 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:00.961208 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":93,"threads_started":1}
I20260809 07:56:01.674573 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:01.674115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:40111)
0809 07:56:01.674389 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:01.674474 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260809 07:56:01.781591 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:01.781192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:42269)
0809 07:56:01.781437 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:01.781492 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
I20260809 07:56:01.962378 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:01.962021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:50779)
0809 07:56:01.962241 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:01.962295 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":70,"threads_started":1}
I20260809 07:56:02.038900 20103 raft_consensus.cc:482] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:56:02.039042 20103 raft_consensus.cc:504] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } }
I20260809 07:56:02.039433 20103 leader_election.cc:288] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 779e66f2e58a4f6ab3e310b66dbade52 (127.28.112.2:32773), 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997)
W20260809 07:56:02.040294 19500 leader_election.cc:341] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 779e66f2e58a4f6ab3e310b66dbade52 (127.28.112.2:32773): Not found: Tablet not found: 1d76d931b9524c1ca4ad2f7b7a2f1891
W20260809 07:56:02.040428 19504 leader_election.cc:341] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997): Not found: Tablet not found: 1d76d931b9524c1ca4ad2f7b7a2f1891
I20260809 07:56:02.040498 19504 leader_election.cc:302] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fbc44e8df044db58a770554ee292de6; no voters: 7113fd4151764215a0c7cada8f56ebcc, 779e66f2e58a4f6ab3e310b66dbade52
I20260809 07:56:02.040645 20103 raft_consensus.cc:2726] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 07:56:02.675702 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:02.675257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:41763)
0809 07:56:02.675541 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:02.675602 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":76,"threads_started":1}
I20260809 07:56:02.782663 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:02.782273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:37327)
0809 07:56:02.782508 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:02.782563 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
I20260809 07:56:02.963425 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:02.963001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:50681)
0809 07:56:02.963242 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:02.963299 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":70,"threads_started":1}
I20260809 07:56:03.676941 20111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:03.676516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:43693)
0809 07:56:03.676795 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:03.676848 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":77,"threads_started":1}
I20260809 07:56:03.783705 20114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:03.783332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:53767)
0809 07:56:03.783558 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:03.783608 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":78,"threads_started":1}
I20260809 07:56:03.964478 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:03.964080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:40267)
0809 07:56:03.964338 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:03.964393 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
I20260809 07:56:03.983634 20116 raft_consensus.cc:482] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:56:03.983757 20116 raft_consensus.cc:504] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } }
I20260809 07:56:03.984130 20116 leader_election.cc:288] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 779e66f2e58a4f6ab3e310b66dbade52 (127.28.112.2:32773), 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997)
W20260809 07:56:03.984853 19504 leader_election.cc:341] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997): Not found: Tablet not found: 1d76d931b9524c1ca4ad2f7b7a2f1891
W20260809 07:56:03.984980 19500 leader_election.cc:341] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 779e66f2e58a4f6ab3e310b66dbade52 (127.28.112.2:32773): Not found: Tablet not found: 1d76d931b9524c1ca4ad2f7b7a2f1891
I20260809 07:56:03.985066 19500 leader_election.cc:302] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fbc44e8df044db58a770554ee292de6; no voters: 7113fd4151764215a0c7cada8f56ebcc, 779e66f2e58a4f6ab3e310b66dbade52
I20260809 07:56:03.985204 20116 raft_consensus.cc:2726] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 07:56:04.678110 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:04.677674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:41209)
0809 07:56:04.677925 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:04.678010 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":95,"threads_started":1}
I20260809 07:56:04.784839 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:04.784401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:56299)
0809 07:56:04.784691 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:04.784744 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":91,"threads_started":1}
I20260809 07:56:04.965618 20123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:04.965149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:47399)
0809 07:56:04.965451 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:04.965518 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
I20260809 07:56:05.679265 20124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:05.678829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:55997)
0809 07:56:05.679090 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:05.679161 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":100,"threads_started":1}
I20260809 07:56:05.785974 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:05.785530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:40911)
0809 07:56:05.785807 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:05.785858 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":68,"threads_started":1}
I20260809 07:56:05.966732 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:05.966321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:56695)
0809 07:56:05.966594 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:05.966647 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":68,"threads_started":1}
I20260809 07:56:06.680491 20132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:06.680059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:44433)
0809 07:56:06.680307 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:06.680363 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
I20260809 07:56:06.787067 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:06.786624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:33989)
0809 07:56:06.786909 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:06.786970 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":70,"threads_started":1}
I20260809 07:56:06.907395 20135 raft_consensus.cc:482] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:56:06.907528 20135 raft_consensus.cc:504] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } }
I20260809 07:56:06.907898 20135 leader_election.cc:288] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 779e66f2e58a4f6ab3e310b66dbade52 (127.28.112.2:32773), 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997)
W20260809 07:56:06.908704 19500 leader_election.cc:341] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 779e66f2e58a4f6ab3e310b66dbade52 (127.28.112.2:32773): Not found: Tablet not found: 1d76d931b9524c1ca4ad2f7b7a2f1891
W20260809 07:56:06.908828 19504 leader_election.cc:341] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997): Not found: Tablet not found: 1d76d931b9524c1ca4ad2f7b7a2f1891
I20260809 07:56:06.908900 19504 leader_election.cc:302] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fbc44e8df044db58a770554ee292de6; no voters: 7113fd4151764215a0c7cada8f56ebcc, 779e66f2e58a4f6ab3e310b66dbade52
I20260809 07:56:06.909035 20135 raft_consensus.cc:2726] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 07:56:06.967796 20137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:06.967386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:34513)
0809 07:56:06.967649 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:06.967705 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":92,"threads_started":1}
I20260809 07:56:07.681586 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:07.681174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:40361)
0809 07:56:07.681433 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:07.681490 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
I20260809 07:56:07.788085 20140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:07.787694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:40611)
0809 07:56:07.787948 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:07.788001 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":91,"threads_started":1}
I20260809 07:56:07.968971 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:07.968582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:49471)
0809 07:56:07.968825 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:07.968879 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260809 07:56:08.682780 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:08.682389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:39957)
0809 07:56:08.682630 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:08.682685 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
I20260809 07:56:08.789131 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:08.788765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:45099)
0809 07:56:08.788993 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:08.789044 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":88,"threads_started":1}
I20260809 07:56:08.970032 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:08.969649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:59377)
0809 07:56:08.969860 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:08.969911 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":69,"threads_started":1}
I20260809 07:56:09.683959 20146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:09.683527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:60573)
0809 07:56:09.683800 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:09.683856 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":81,"threads_started":1}
I20260809 07:56:09.790220 20147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:09.789802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:37783)
0809 07:56:09.790064 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:09.790116 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":83,"threads_started":1}
I20260809 07:56:09.971163 20148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:09.970710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:47475)
0809 07:56:09.970976 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:09.971040 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
I20260809 07:56:10.685153 20149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:10.684686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:52273)
0809 07:56:10.684968 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:10.685049 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":82,"threads_started":1}
I20260809 07:56:10.791301 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:10.790890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:49041)
0809 07:56:10.791150 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:10.791203 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
I20260809 07:56:10.842193 20155 raft_consensus.cc:482] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:56:10.842336 20155 raft_consensus.cc:504] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } }
I20260809 07:56:10.842760 20155 leader_election.cc:288] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 779e66f2e58a4f6ab3e310b66dbade52 (127.28.112.2:32773), 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997)
W20260809 07:56:10.843540 19504 leader_election.cc:341] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997): Not found: Tablet not found: 1d76d931b9524c1ca4ad2f7b7a2f1891
W20260809 07:56:10.843654 19500 leader_election.cc:341] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 779e66f2e58a4f6ab3e310b66dbade52 (127.28.112.2:32773): Not found: Tablet not found: 1d76d931b9524c1ca4ad2f7b7a2f1891
I20260809 07:56:10.843720 19500 leader_election.cc:302] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fbc44e8df044db58a770554ee292de6; no voters: 7113fd4151764215a0c7cada8f56ebcc, 779e66f2e58a4f6ab3e310b66dbade52
I20260809 07:56:10.843888 20155 raft_consensus.cc:2726] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 07:56:10.972268 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:10.971857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:44307)
0809 07:56:10.972114 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:10.972178 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":108,"threads_started":1}
I20260809 07:56:11.686368 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:11.685883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:60537)
0809 07:56:11.686180 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:11.686247 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":111,"threads_started":1}
I20260809 07:56:11.792368 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:11.792001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:58355)
0809 07:56:11.792220 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:11.792274 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":71,"threads_started":1}
I20260809 07:56:11.973405 20159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:11.972963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:54077)
0809 07:56:11.973250 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:11.973309 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":95,"threads_started":1}
I20260809 07:56:12.687515 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:12.687085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:37091)
0809 07:56:12.687362 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:12.687417 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":98,"threads_started":1}
I20260809 07:56:12.793429 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:12.793043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:36589)
0809 07:56:12.793289 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:12.793340 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":99,"threads_started":1}
I20260809 07:56:12.974469 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:12.974077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:38149)
0809 07:56:12.974321 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:12.974374 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":84,"threads_started":1}
I20260809 07:56:13.688889 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:13.688450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.1:44413)
0809 07:56:13.688708 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:13.688768 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
I20260809 07:56:13.794673 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:13.794248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.2:38633)
0809 07:56:13.794521 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:13.794576 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":90,"threads_started":1}
I20260809 07:56:13.907025 20166 raft_consensus.cc:482] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:56:13.907186 20166 raft_consensus.cc:504] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779e66f2e58a4f6ab3e310b66dbade52" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 32773 } } peers { permanent_uuid: "7113fd4151764215a0c7cada8f56ebcc" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 39997 } } peers { permanent_uuid: "1fbc44e8df044db58a770554ee292de6" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35403 } }
I20260809 07:56:13.907631 20166 leader_election.cc:288] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 779e66f2e58a4f6ab3e310b66dbade52 (127.28.112.2:32773), 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997)
W20260809 07:56:13.908452 19504 leader_election.cc:341] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7113fd4151764215a0c7cada8f56ebcc (127.28.112.3:39997): Not found: Tablet not found: 1d76d931b9524c1ca4ad2f7b7a2f1891
W20260809 07:56:13.908555 19500 leader_election.cc:341] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 779e66f2e58a4f6ab3e310b66dbade52 (127.28.112.2:32773): Not found: Tablet not found: 1d76d931b9524c1ca4ad2f7b7a2f1891
I20260809 07:56:13.908617 19500 leader_election.cc:302] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fbc44e8df044db58a770554ee292de6; no voters: 7113fd4151764215a0c7cada8f56ebcc, 779e66f2e58a4f6ab3e310b66dbade52
I20260809 07:56:13.908761 20166 raft_consensus.cc:2726] T 1d76d931b9524c1ca4ad2f7b7a2f1891 P 1fbc44e8df044db58a770554ee292de6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 07:56:13.958976 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 19429
I20260809 07:56:13.975224 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 19683
I20260809 07:56:13.978766 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:13.975262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:43185 (local address 127.28.112.3:42463)
0809 07:56:13.978585 (+ 3323us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:13.978647 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3237,"thread_start_us":62,"threads_started":1}
I20260809 07:56:13.985517 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 19894
2026-08-09T07:56:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19267 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:56:14 dist-test-slave-vpvm krb5kdc[20172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:56:14 dist-test-slave-vpvm krb5kdc[20172](info): set up 2 sockets
Aug 09 07:56:14 dist-test-slave-vpvm krb5kdc[20172](info): commencing operation
krb5kdc: starting...
W20260809 07:56:15.138180 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:56:15 dist-test-slave-vpvm krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:56:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:56:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:56:15.252719 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:36865
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:36865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:56:15.363278 20190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:56:15.363547 20190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:56:15.363588 20190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:56:15.367728 20190 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 07:56:15.367779 20190 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 07:56:15.367795 20190 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 07:56:15.367812 20190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:56:15.367851 20190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:56:15.367887 20190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:56:15.367929 20190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:56:15.367964 20190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:56:15.373056 20190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:36865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:36865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:56:15.374061 20190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:56:15.375077 20190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:56:15.383249 20195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:56:15.383297 20190 server_base.cc:1034] running on GCE node
W20260809 07:56:15.383266 20197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:56:15.383239 20199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:56:15.383996 20190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:56:15.385010 20190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:56:15.386173 20190 hybrid_clock.cc:648] HybridClock initialized: now 1786262175386152 us; error 38 us; skew 500 ppm
Aug 09 07:56:15 dist-test-slave-vpvm krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262175, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:56:15.388782 20190 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:56:15.389624 20190 webserver.cc:460] Webserver started at http://127.28.112.62:40361/ using document root <none> and password file <none>
I20260809 07:56:15.390172 20190 fs_manager.cc:359] Metadata directory not provided
I20260809 07:56:15.390239 20190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:56:15.390435 20190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:56:15.392364 20190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "1739d096f8894e84ad0fd97c64277848"
format_stamp: "Formatted at 2026-08-09 07:56:15 on dist-test-slave-vpvm"
server_key: "7cc3dce32ca20324dbd7be418679d6f9"
server_key_iv: "34b70f224d5ea96ad18e2ed3f8bb869d"
server_key_version: "encryptionkey@0"
I20260809 07:56:15.392892 20190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "1739d096f8894e84ad0fd97c64277848"
format_stamp: "Formatted at 2026-08-09 07:56:15 on dist-test-slave-vpvm"
server_key: "7cc3dce32ca20324dbd7be418679d6f9"
server_key_iv: "34b70f224d5ea96ad18e2ed3f8bb869d"
server_key_version: "encryptionkey@0"
I20260809 07:56:15.396401 20190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 07:56:15.398716 20209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:15.399871 20190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:56:15.399986 20190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "1739d096f8894e84ad0fd97c64277848"
format_stamp: "Formatted at 2026-08-09 07:56:15 on dist-test-slave-vpvm"
server_key: "7cc3dce32ca20324dbd7be418679d6f9"
server_key_iv: "34b70f224d5ea96ad18e2ed3f8bb869d"
server_key_version: "encryptionkey@0"
I20260809 07:56:15.400072 20190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:56:15.416762 20190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:56:15.419404 20190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:56:15.419572 20190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:56:15.427858 20190 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:36865
I20260809 07:56:15.428141 20289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:36865 every 8 connection(s)
I20260809 07:56:15.428836 20190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:56:15.431502 20291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:56:15.436709 20291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848: Bootstrap starting.
I20260809 07:56:15.438310 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 20190
I20260809 07:56:15.438441 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:56:15.438751 29120 external_mini_cluster.cc:1428] Setting key 56e9f6c90688290ef1fd946bac53fcd3
I20260809 07:56:15.439298 20291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848: Neither blocks nor log segments found. Creating new log.
I20260809 07:56:15.440140 20291 log.cc:824] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848: Log is configured to *not* fsync() on all Append() calls
I20260809 07:56:15.441881 20291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848: No bootstrap required, opened a new log
I20260809 07:56:15.445117 20291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1739d096f8894e84ad0fd97c64277848" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 36865 } }
I20260809 07:56:15.445376 20291 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:56:15.445443 20291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1739d096f8894e84ad0fd97c64277848, State: Initialized, Role: FOLLOWER
I20260809 07:56:15.445859 20291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1739d096f8894e84ad0fd97c64277848" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 36865 } }
I20260809 07:56:15.446005 20291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:56:15.446079 20291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:56:15.446171 20291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 0 FOLLOWER]: Advancing to term 1
Aug 09 07:56:15 dist-test-slave-vpvm krb5kdc[20172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:56:15.447142 20291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1739d096f8894e84ad0fd97c64277848" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 36865 } }
I20260809 07:56:15.447448 20291 leader_election.cc:302] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1739d096f8894e84ad0fd97c64277848; no voters:
I20260809 07:56:15.447724 20291 leader_election.cc:288] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:56:15.447803 20300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:56:15.448043 20300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 1 LEADER]: Becoming Leader. State: Replica: 1739d096f8894e84ad0fd97c64277848, State: Running, Role: LEADER
I20260809 07:56:15.448364 20300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1739d096f8894e84ad0fd97c64277848" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 36865 } }
I20260809 07:56:15.448892 20291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:56:15.449990 20303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1739d096f8894e84ad0fd97c64277848. Latest consensus state: current_term: 1 leader_uuid: "1739d096f8894e84ad0fd97c64277848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1739d096f8894e84ad0fd97c64277848" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 36865 } } }
I20260809 07:56:15.450115 20303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [sys.catalog]: This master's current role is: LEADER
I20260809 07:56:15.450426 20302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1739d096f8894e84ad0fd97c64277848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1739d096f8894e84ad0fd97c64277848" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 36865 } } }
I20260809 07:56:15.450562 20302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [sys.catalog]: This master's current role is: LEADER
I20260809 07:56:15.450498 20312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:56:15.452699 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:15.439827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44658 (local address 127.28.112.62:36865)
0809 07:56:15.440253 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:15.440272 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:15.440313 (+ 41us) server_negotiation.cc:407] Connection header received
0809 07:56:15.441020 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:15.441041 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:15.441356 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:15.441647 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:56:15.442470 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:15.443434 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:15.444258 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:15.444701 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:15.444847 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:15.447373 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:56:15.447378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:56:15.447390 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:56:15.447417 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:56:15.449502 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:15.449995 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:15.450002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:15.450008 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:15.450143 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:15.450468 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:15.450472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:15.450473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:15.450974 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:56:15.451160 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:15.451594 (+ 434us) server_negotiation.cc:299] Negotiation successful
0809 07:56:15.451877 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":130,"threads_started":1}
I20260809 07:56:15.453907 20312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:56:15.458997 20312 catalog_manager.cc:1347] Generated new cluster ID: 2ffe21266a6d42b89bac0f6cd98c05a2
I20260809 07:56:15.459064 20312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:56:15.465083 20312 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:56:15.465875 20312 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:56:15.472023 20312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848: Generated new TSK 0
I20260809 07:56:15.472697 20312 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
I20260809 07:56:15.503178 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:36865
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:56:15.611809 20333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:56:15.612047 20333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:56:15.612104 20333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:56:15.615963 20333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:56:15.616014 20333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:56:15.616089 20333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:56:15.621012 20333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:56:15.621932 20333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:56:15.622910 20333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:56:15.632072 20344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:56:15.632068 20341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:56:15.632076 20340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:56:15.632295 20333 server_base.cc:1034] running on GCE node
I20260809 07:56:15.632750 20333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:56:15.633329 20333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:56:15.634490 20333 hybrid_clock.cc:648] HybridClock initialized: now 1786262175634472 us; error 28 us; skew 500 ppm
Aug 09 07:56:15 dist-test-slave-vpvm krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262175, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:56:15.636967 20333 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:56:15.637794 20333 webserver.cc:460] Webserver started at http://127.28.112.1:34849/ using document root <none> and password file <none>
I20260809 07:56:15.638339 20333 fs_manager.cc:359] Metadata directory not provided
I20260809 07:56:15.638406 20333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:56:15.638633 20333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:56:15.640537 20333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "5d1d92bb8bd84610b33f8546b515e0fd"
format_stamp: "Formatted at 2026-08-09 07:56:15 on dist-test-slave-vpvm"
server_key: "ca11be9f18842afb6a50b0f156aea67d"
server_key_iv: "244f05d4e9b2226a6a8edfaf93d7d69a"
server_key_version: "encryptionkey@0"
I20260809 07:56:15.641050 20333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "5d1d92bb8bd84610b33f8546b515e0fd"
format_stamp: "Formatted at 2026-08-09 07:56:15 on dist-test-slave-vpvm"
server_key: "ca11be9f18842afb6a50b0f156aea67d"
server_key_iv: "244f05d4e9b2226a6a8edfaf93d7d69a"
server_key_version: "encryptionkey@0"
I20260809 07:56:15.644488 20333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 07:56:15.646684 20355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:15.647756 20333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:56:15.647858 20333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "5d1d92bb8bd84610b33f8546b515e0fd"
format_stamp: "Formatted at 2026-08-09 07:56:15 on dist-test-slave-vpvm"
server_key: "ca11be9f18842afb6a50b0f156aea67d"
server_key_iv: "244f05d4e9b2226a6a8edfaf93d7d69a"
server_key_version: "encryptionkey@0"
I20260809 07:56:15.647945 20333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:56:15.670286 20333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:56:15.673107 20333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:56:15.673266 20333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:56:15.673806 20333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:56:15.674606 20333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:56:15.674654 20333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:15.674701 20333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:56:15.674731 20333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:15.682727 20333 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:39299
I20260809 07:56:15.682775 20521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:39299 every 8 connection(s)
I20260809 07:56:15.683550 20333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:56:15.689532 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 20333
I20260809 07:56:15.689659 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:56:15.689991 29120 external_mini_cluster.cc:1428] Setting key e03b94b532ae00d1407a9adb7c848c57
Aug 09 07:56:15 dist-test-slave-vpvm krb5kdc[20172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262175, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:56:15.696758 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:15.685210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:49277 (local address 127.28.112.62:36865)
0809 07:56:15.685354 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:15.685359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:15.686220 (+ 861us) server_negotiation.cc:407] Connection header received
0809 07:56:15.687505 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:15.687508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:15.687555 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:15.687638 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:56:15.688694 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:15.689221 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:15.690711 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:15.690951 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:15.691047 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:15.693525 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:56:15.693529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:56:15.693532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:56:15.693555 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:56:15.694852 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:15.695322 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:15.695325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:15.695327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:15.695402 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:15.695711 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:15.695714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:15.695716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:15.695890 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:56:15.695989 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:15.696472 (+ 483us) server_negotiation.cc:299] Negotiation successful
0809 07:56:15.696600 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 07:56:15.697477 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:15.685566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:49277)
0809 07:56:15.686068 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:15.686119 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:15.687296 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:15.687797 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:15.687803 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:15.688123 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:56:15.688568 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:15.688581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:15.689354 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:15.689357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:15.690586 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:15.690598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:15.691113 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:15.691117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:15.691308 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:15.691850 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:56:15.691866 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:56:15.693366 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:56:15.694964 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:15.694971 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:15.694985 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:15.695226 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:15.695491 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:15.695494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:15.695496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:15.695624 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:15.696000 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:56:15.696007 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:56:15.696243 (+ 236us) client_negotiation.cc:769] Sending connection context
0809 07:56:15.696402 (+ 159us) client_negotiation.cc:240] Negotiation successful
0809 07:56:15.696691 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":132,"threads_started":1}
I20260809 07:56:15.698654 20523 heartbeater.cc:347] Connected to a master server at 127.28.112.62:36865
I20260809 07:56:15.698907 20523 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.112.2@KRBTEST.COM; defaulting to no policy
I20260809 07:56:15.699473 20523 heartbeater.cc:511] Master 127.28.112.62:36865 requested a full tablet report, sending...
I20260809 07:56:15.700984 20235 ts_manager.cc:194] Registered new tserver with Master: 5d1d92bb8bd84610b33f8546b515e0fd (127.28.112.1:39299)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.2@KRBTEST.COM" created.
I20260809 07:56:15.702175 20235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:49277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
I20260809 07:56:15.716657 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:36865
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:56:15.835892 20531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:56:15.836139 20531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:56:15.836179 20531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:56:15.840137 20531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:56:15.840198 20531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:56:15.840255 20531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:56:15.845393 20531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:56:15.846480 20531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:56:15.847473 20531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:56:15.856943 20538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:56:15.857051 20531 server_base.cc:1034] running on GCE node
W20260809 07:56:15.856951 20541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:56:15.856969 20543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:56:15.857592 20531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:56:15.858256 20531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:56:15.859401 20531 hybrid_clock.cc:648] HybridClock initialized: now 1786262175859376 us; error 38 us; skew 500 ppm
Aug 09 07:56:15 dist-test-slave-vpvm krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262175, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:56:15.862051 20531 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:56:15.862896 20531 webserver.cc:460] Webserver started at http://127.28.112.2:34437/ using document root <none> and password file <none>
I20260809 07:56:15.863440 20531 fs_manager.cc:359] Metadata directory not provided
I20260809 07:56:15.863500 20531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:56:15.863689 20531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:56:15.865628 20531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "f9609c8bb0354680b799dd060579a25a"
format_stamp: "Formatted at 2026-08-09 07:56:15 on dist-test-slave-vpvm"
server_key: "6e0c8c8a004a15ab67bc208e93e20604"
server_key_iv: "0f24c2c132a926f32ef99e99800aa770"
server_key_version: "encryptionkey@0"
I20260809 07:56:15.866195 20531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "f9609c8bb0354680b799dd060579a25a"
format_stamp: "Formatted at 2026-08-09 07:56:15 on dist-test-slave-vpvm"
server_key: "6e0c8c8a004a15ab67bc208e93e20604"
server_key_iv: "0f24c2c132a926f32ef99e99800aa770"
server_key_version: "encryptionkey@0"
I20260809 07:56:15.870179 20531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 07:56:15.872586 20557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:15.873817 20531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:56:15.873934 20531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "f9609c8bb0354680b799dd060579a25a"
format_stamp: "Formatted at 2026-08-09 07:56:15 on dist-test-slave-vpvm"
server_key: "6e0c8c8a004a15ab67bc208e93e20604"
server_key_iv: "0f24c2c132a926f32ef99e99800aa770"
server_key_version: "encryptionkey@0"
I20260809 07:56:15.874078 20531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:56:15.885651 20531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:56:15.888357 20531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:56:15.888535 20531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:56:15.889103 20531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:56:15.889899 20531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:56:15.889966 20531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:15.890021 20531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:56:15.890053 20531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:15.898824 20531 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:46401
I20260809 07:56:15.898865 20734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:46401 every 8 connection(s)
I20260809 07:56:15.899688 20531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:56:15.902765 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 20531
I20260809 07:56:15.902905 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:56:15.903271 29120 external_mini_cluster.cc:1428] Setting key 4426a6a02a603f814d960aa4b9c82c2e
Aug 09 07:56:15 dist-test-slave-vpvm krb5kdc[20172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262175, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.112.3@KRBTEST.COM; defaulting to no policy
I20260809 07:56:15.913337 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:15.901451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:58637 (local address 127.28.112.62:36865)
0809 07:56:15.901625 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:15.901628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:15.902489 (+ 861us) server_negotiation.cc:407] Connection header received
0809 07:56:15.904010 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:15.904013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:15.904093 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:15.904187 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:56:15.905525 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:15.906117 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:15.907261 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:15.907500 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:15.907595 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:15.910136 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:56:15.910140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:56:15.910143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:56:15.910166 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:56:15.911422 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:15.911898 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:15.911901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:15.911902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:15.911977 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:15.912294 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:15.912297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:15.912299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:15.912469 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:56:15.912556 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:15.913053 (+ 497us) server_negotiation.cc:299] Negotiation successful
0809 07:56:15.913170 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":76}
I20260809 07:56:15.914201 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:15.901823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:58637)
0809 07:56:15.902320 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:15.902363 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:15.903817 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:15.904406 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:15.904417 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:15.904880 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:56:15.905367 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:15.905382 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:15.906242 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:15.906247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:15.907131 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:15.907141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:15.907655 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:15.907660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:15.907831 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:15.908403 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:56:15.908423 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:56:15.909981 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:56:15.911534 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:15.911540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:15.911554 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:15.911801 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:15.912061 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:15.912064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:15.912066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:15.912205 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:15.912570 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:56:15.912576 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:56:15.912806 (+ 230us) client_negotiation.cc:769] Sending connection context
0809 07:56:15.912981 (+ 175us) client_negotiation.cc:240] Negotiation successful
0809 07:56:15.913300 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":156,"threads_started":1}
I20260809 07:56:15.915206 20735 heartbeater.cc:347] Connected to a master server at 127.28.112.62:36865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.3@KRBTEST.COM" created.
I20260809 07:56:15.915489 20735 heartbeater.cc:464] Registering TS with master...
I20260809 07:56:15.915993 20735 heartbeater.cc:511] Master 127.28.112.62:36865 requested a full tablet report, sending...
I20260809 07:56:15.917068 20235 ts_manager.cc:194] Registered new tserver with Master: f9609c8bb0354680b799dd060579a25a (127.28.112.2:46401)
I20260809 07:56:15.917620 20235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:58637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
I20260809 07:56:15.929576 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:36865
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:56:16.038679 20741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:56:16.038914 20741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:56:16.038972 20741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:56:16.042907 20741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:56:16.042961 20741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:56:16.043035 20741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:56:16.048010 20741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:56:16.048954 20741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:56:16.049940 20741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:56:16.059088 20748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:56:16.059091 20749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:56:16.059160 20741 server_base.cc:1034] running on GCE node
W20260809 07:56:16.059300 20751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:56:16.059659 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:56:16.060256 20741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:56:16.061404 20741 hybrid_clock.cc:648] HybridClock initialized: now 1786262176061391 us; error 54 us; skew 500 ppm
Aug 09 07:56:16 dist-test-slave-vpvm krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262176, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:56:16.064018 20741 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:56:16.064850 20741 webserver.cc:460] Webserver started at http://127.28.112.3:42599/ using document root <none> and password file <none>
I20260809 07:56:16.065382 20741 fs_manager.cc:359] Metadata directory not provided
I20260809 07:56:16.065447 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:56:16.065624 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:56:16.067585 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "79ece8ecd514451abbcebc8b74a37bf5"
format_stamp: "Formatted at 2026-08-09 07:56:16 on dist-test-slave-vpvm"
server_key: "d75d76a517a5da229392abab36a4fbb4"
server_key_iv: "a72e5f8a5bdecffd53b208bd2614e8b3"
server_key_version: "encryptionkey@0"
I20260809 07:56:16.068104 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "79ece8ecd514451abbcebc8b74a37bf5"
format_stamp: "Formatted at 2026-08-09 07:56:16 on dist-test-slave-vpvm"
server_key: "d75d76a517a5da229392abab36a4fbb4"
server_key_iv: "a72e5f8a5bdecffd53b208bd2614e8b3"
server_key_version: "encryptionkey@0"
I20260809 07:56:16.071659 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260809 07:56:16.073967 20763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:16.075173 20741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 07:56:16.075280 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "79ece8ecd514451abbcebc8b74a37bf5"
format_stamp: "Formatted at 2026-08-09 07:56:16 on dist-test-slave-vpvm"
server_key: "d75d76a517a5da229392abab36a4fbb4"
server_key_iv: "a72e5f8a5bdecffd53b208bd2614e8b3"
server_key_version: "encryptionkey@0"
I20260809 07:56:16.075376 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:56:16.088027 20741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:56:16.090795 20741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:56:16.090960 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:56:16.091497 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:56:16.092271 20741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:56:16.092329 20741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:16.092382 20741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:56:16.092410 20741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:16.101001 20741 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:44481
I20260809 07:56:16.101078 20936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:44481 every 8 connection(s)
I20260809 07:56:16.101836 20741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:56:16.104924 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 20741
I20260809 07:56:16.105026 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:56:16.105279 29120 external_mini_cluster.cc:1428] Setting key fd775c8f3d8ff008b9b881811c8ed19e
Aug 09 07:56:16 dist-test-slave-vpvm krb5kdc[20172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262176, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:56:16.114702 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:16.103552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:35137 (local address 127.28.112.62:36865)
0809 07:56:16.103697 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:16.103701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:16.104517 (+ 816us) server_negotiation.cc:407] Connection header received
0809 07:56:16.105878 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:16.105882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:16.105939 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:16.106042 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:56:16.107279 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.107829 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.109006 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.109224 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.109293 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:16.111622 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:56:16.111625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:56:16.111627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:56:16.111659 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:56:16.112961 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:16.113408 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:16.113410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:16.113411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:16.113479 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:16.113761 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:16.113763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:16.113764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:16.113922 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:56:16.114032 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:16.114474 (+ 442us) server_negotiation.cc:299] Negotiation successful
0809 07:56:16.114576 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 07:56:16.115511 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:16.103925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:35137)
0809 07:56:16.104367 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:16.104407 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:16.105659 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:16.106244 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:16.106251 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:16.106656 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:56:16.107133 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:16.107150 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.107974 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.107978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:16.108879 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:16.108890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.109372 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.109375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:16.109528 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:16.110072 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:56:16.110088 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:56:16.111464 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:56:16.113071 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:16.113076 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:16.113088 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:16.113327 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:16.113556 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:16.113559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:16.113560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:16.113690 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:16.114045 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:56:16.114051 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:56:16.114275 (+ 224us) client_negotiation.cc:769] Sending connection context
0809 07:56:16.114450 (+ 175us) client_negotiation.cc:240] Negotiation successful
0809 07:56:16.114696 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":127,"threads_started":1}
I20260809 07:56:16.116554 20937 heartbeater.cc:347] Connected to a master server at 127.28.112.62:36865
I20260809 07:56:16.116763 20937 heartbeater.cc:464] Registering TS with master...
I20260809 07:56:16.117219 20937 heartbeater.cc:511] Master 127.28.112.62:36865 requested a full tablet report, sending...
I20260809 07:56:16.118345 20235 ts_manager.cc:194] Registered new tserver with Master: 79ece8ecd514451abbcebc8b74a37bf5 (127.28.112.3:44481)
I20260809 07:56:16.119094 20235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:35137
I20260809 07:56:16.119659 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 07:56:16.127981 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:16.121827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44664 (local address 127.28.112.62:36865)
0809 07:56:16.121932 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:16.121937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:16.122137 (+ 200us) server_negotiation.cc:407] Connection header received
0809 07:56:16.122223 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:16.122225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:16.122260 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:16.122323 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:56:16.122907 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.123397 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.124115 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.124308 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.124373 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:16.125479 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:56:16.125482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:56:16.125484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:56:16.125504 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:56:16.126805 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:16.127133 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:16.127136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:16.127137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:16.127200 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:16.127373 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:16.127374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:16.127375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:16.127556 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:56:16.127664 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:16.127789 (+ 125us) server_negotiation.cc:299] Negotiation successful
0809 07:56:16.127873 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 07:56:16.130563 20233 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:56:16.133395 20233 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:56:16.146697 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:16.141104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:46401 (local address 127.0.0.1:60980)
0809 07:56:16.141771 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:16.141821 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:16.141943 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:16.142757 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:16.142763 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:16.142783 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:56:16.142911 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:16.142918 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.144008 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.144012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:16.145145 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:16.145156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.146201 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.146205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:16.146307 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:16.146342 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 07:56:16.146441 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 07:56:16.146566 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":76,"threads_started":1}
I20260809 07:56:16.146711 20956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:16.141273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60980 (local address 127.28.112.2:46401)
0809 07:56:16.142272 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:16.142284 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:16.142310 (+ 26us) server_negotiation.cc:407] Connection header received
0809 07:56:16.142390 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:16.142396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:16.142603 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:16.142756 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:56:16.143044 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.143883 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.145283 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.146093 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.146189 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:16.146331 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:16.146479 (+ 148us) server_negotiation.cc:299] Negotiation successful
0809 07:56:16.146567 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":892,"thread_start_us":114,"threads_started":1}
I20260809 07:56:16.147195 20957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:16.141455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:44481 (local address 127.0.0.1:54568)
0809 07:56:16.142332 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:16.142345 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:16.142423 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:16.143129 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:16.143132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:16.143164 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:56:16.143245 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:16.143251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.144529 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.144532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:16.145572 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:16.145582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.146808 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.146811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:16.146910 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:16.146932 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 07:56:16.146999 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 07:56:16.147065 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":49,"thread_start_us":78,"threads_started":1}
I20260809 07:56:16.147315 20952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:16.141212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:39299 (local address 127.0.0.1:48402)
0809 07:56:16.142109 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:16.142121 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:16.142203 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:16.142937 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:16.142940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:16.142957 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:56:16.143055 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:16.143062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.144500 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.144504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:16.145658 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:16.145668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.146972 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.146975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:16.147060 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:16.147080 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 07:56:16.147151 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 07:56:16.147207 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"thread_start_us":88,"threads_started":1}
I20260809 07:56:16.147670 20959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:16.141880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54568 (local address 127.28.112.3:44481)
0809 07:56:16.142659 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:16.142676 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:16.142698 (+ 22us) server_negotiation.cc:407] Connection header received
0809 07:56:16.142765 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:16.142770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:16.142938 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:16.143074 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:56:16.143354 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.144417 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.145692 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.146490 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.146595 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:16.146743 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:16.147435 (+ 692us) server_negotiation.cc:299] Negotiation successful
0809 07:56:16.147524 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":146,"threads_started":1}
I20260809 07:56:16.147667 20958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:16.141872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48402 (local address 127.28.112.1:39299)
0809 07:56:16.142526 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:16.142540 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:16.142562 (+ 22us) server_negotiation.cc:407] Connection header received
0809 07:56:16.142636 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:16.142641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:16.142805 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:16.142945 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:56:16.143210 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.144351 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.145772 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.146695 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.146793 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:16.146914 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:16.147464 (+ 550us) server_negotiation.cc:299] Negotiation successful
0809 07:56:16.147539 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":140,"threads_started":1}
I20260809 07:56:16.150059 20630 tablet_service.cc:1467] Processing CreateTablet for tablet 6b7f2044be67427b81d023e28f131c7a (DEFAULT_TABLE table=test-table [id=9f281f3ce6d94c2d9a409de3ec4a9194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:56:16.150326 20843 tablet_service.cc:1467] Processing CreateTablet for tablet 6b7f2044be67427b81d023e28f131c7a (DEFAULT_TABLE table=test-table [id=9f281f3ce6d94c2d9a409de3ec4a9194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:56:16.150326 20428 tablet_service.cc:1467] Processing CreateTablet for tablet 6b7f2044be67427b81d023e28f131c7a (DEFAULT_TABLE table=test-table [id=9f281f3ce6d94c2d9a409de3ec4a9194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:56:16.151129 20630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b7f2044be67427b81d023e28f131c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:56:16.151341 20428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b7f2044be67427b81d023e28f131c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:56:16.151346 20843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b7f2044be67427b81d023e28f131c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:56:16.156980 20963 tablet_bootstrap.cc:492] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a: Bootstrap starting.
I20260809 07:56:16.158394 20964 tablet_bootstrap.cc:492] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd: Bootstrap starting.
I20260809 07:56:16.158979 20965 tablet_bootstrap.cc:492] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5: Bootstrap starting.
I20260809 07:56:16.159797 20963 tablet_bootstrap.cc:654] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a: Neither blocks nor log segments found. Creating new log.
I20260809 07:56:16.160552 20964 tablet_bootstrap.cc:654] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd: Neither blocks nor log segments found. Creating new log.
I20260809 07:56:16.160674 20963 log.cc:824] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a: Log is configured to *not* fsync() on all Append() calls
I20260809 07:56:16.160962 20965 tablet_bootstrap.cc:654] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5: Neither blocks nor log segments found. Creating new log.
I20260809 07:56:16.161731 20965 log.cc:824] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5: Log is configured to *not* fsync() on all Append() calls
I20260809 07:56:16.161767 20964 log.cc:824] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd: Log is configured to *not* fsync() on all Append() calls
I20260809 07:56:16.162957 20963 tablet_bootstrap.cc:492] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a: No bootstrap required, opened a new log
I20260809 07:56:16.163154 20963 ts_tablet_manager.cc:1397] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260809 07:56:16.163666 20964 tablet_bootstrap.cc:492] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd: No bootstrap required, opened a new log
I20260809 07:56:16.163667 20965 tablet_bootstrap.cc:492] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5: No bootstrap required, opened a new log
I20260809 07:56:16.163848 20965 ts_tablet_manager.cc:1397] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 07:56:16.163848 20964 ts_tablet_manager.cc:1397] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 07:56:16.165899 20963 raft_consensus.cc:348] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } } peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } }
I20260809 07:56:16.166275 20963 raft_consensus.cc:374] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:56:16.166381 20963 raft_consensus.cc:729] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9609c8bb0354680b799dd060579a25a, State: Initialized, Role: FOLLOWER
I20260809 07:56:16.166924 20963 consensus_queue.cc:260] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } } peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } }
I20260809 07:56:16.167557 20735 heartbeater.cc:503] Master 127.28.112.62:36865 was elected leader, sending a full tablet report...
I20260809 07:56:16.167482 20965 raft_consensus.cc:348] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } } peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } }
I20260809 07:56:16.167747 20965 raft_consensus.cc:374] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:56:16.167800 20965 raft_consensus.cc:729] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ece8ecd514451abbcebc8b74a37bf5, State: Initialized, Role: FOLLOWER
I20260809 07:56:16.167874 20963 ts_tablet_manager.cc:1428] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 07:56:16.168190 20965 consensus_queue.cc:260] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } } peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } }
I20260809 07:56:16.168164 20964 raft_consensus.cc:348] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } } peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } }
I20260809 07:56:16.168347 20964 raft_consensus.cc:374] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:56:16.168426 20964 raft_consensus.cc:729] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1d92bb8bd84610b33f8546b515e0fd, State: Initialized, Role: FOLLOWER
I20260809 07:56:16.168805 20937 heartbeater.cc:503] Master 127.28.112.62:36865 was elected leader, sending a full tablet report...
I20260809 07:56:16.169046 20965 ts_tablet_manager.cc:1428] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 07:56:16.169817 20964 consensus_queue.cc:260] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } } peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } }
I20260809 07:56:16.170719 20523 heartbeater.cc:503] Master 127.28.112.62:36865 was elected leader, sending a full tablet report...
I20260809 07:56:16.170778 20964 ts_tablet_manager.cc:1428] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
W20260809 07:56:16.184527 20524 tablet.cc:2369] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:56:16.314436 20973 raft_consensus.cc:482] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:56:16.314688 20973 raft_consensus.cc:504] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } } peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } }
I20260809 07:56:16.315868 20973 leader_election.cc:288] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d1d92bb8bd84610b33f8546b515e0fd (127.28.112.1:39299), 79ece8ecd514451abbcebc8b74a37bf5 (127.28.112.3:44481)
I20260809 07:56:16.320446 20959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:16.316266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:37121 (local address 127.28.112.3:44481)
0809 07:56:16.316415 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:16.316420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:16.316757 (+ 337us) server_negotiation.cc:407] Connection header received
0809 07:56:16.316868 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:16.316870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:16.316909 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:16.316982 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:56:16.317200 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.317885 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.319256 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.319858 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.319978 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:16.320033 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:16.320244 (+ 211us) server_negotiation.cc:299] Negotiation successful
0809 07:56:16.320312 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 07:56:16.320446 20958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:16.316268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:35577 (local address 127.28.112.1:39299)
0809 07:56:16.316434 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:16.316439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:16.316456 (+ 17us) server_negotiation.cc:407] Connection header received
0809 07:56:16.316542 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:16.316545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:16.316596 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:16.316666 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:56:16.316971 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.317727 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.319258 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.319858 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.319978 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:16.320033 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:16.320244 (+ 211us) server_negotiation.cc:299] Negotiation successful
0809 07:56:16.320312 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 07:56:16.320477 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:16.316152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:39299 (local address 127.28.112.2:35577)
0809 07:56:16.316291 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:16.316308 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:16.316406 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:16.316706 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:16.316708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:16.316731 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:56:16.316867 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:16.316874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.317831 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.317834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:16.319113 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:16.319125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.319965 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.319969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:16.320111 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:16.320148 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 07:56:16.320224 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 07:56:16.320315 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":54,"spinlock_wait_cycles":19200}
I20260809 07:56:16.320485 20979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:16.316159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:44481 (local address 127.28.112.2:37121)
0809 07:56:16.316653 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:16.316667 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:16.316784 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:16.317000 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:16.317002 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:16.317019 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:56:16.317102 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:16.317106 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.318005 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.318008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:16.319166 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:16.319174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.319965 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.319969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:16.320144 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:16.320163 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 07:56:16.320224 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 07:56:16.320315 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":189,"spinlock_wait_cycles":1536,"thread_start_us":134,"threads_started":1}
I20260809 07:56:16.321206 20870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b7f2044be67427b81d023e28f131c7a" candidate_uuid: "f9609c8bb0354680b799dd060579a25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ece8ecd514451abbcebc8b74a37bf5" is_pre_election: true
I20260809 07:56:16.321205 20457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b7f2044be67427b81d023e28f131c7a" candidate_uuid: "f9609c8bb0354680b799dd060579a25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" is_pre_election: true
I20260809 07:56:16.321518 20457 raft_consensus.cc:2454] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9609c8bb0354680b799dd060579a25a in term 0.
I20260809 07:56:16.321518 20870 raft_consensus.cc:2454] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9609c8bb0354680b799dd060579a25a in term 0.
I20260809 07:56:16.322041 20559 leader_election.cc:302] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d1d92bb8bd84610b33f8546b515e0fd, f9609c8bb0354680b799dd060579a25a; no voters:
I20260809 07:56:16.322306 20973 raft_consensus.cc:2781] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:56:16.322392 20973 raft_consensus.cc:482] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:56:16.322441 20973 raft_consensus.cc:3037] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:56:16.323487 20973 raft_consensus.cc:504] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } } peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } }
I20260809 07:56:16.323840 20973 leader_election.cc:288] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [CANDIDATE]: Term 1 election: Requested vote from peers 5d1d92bb8bd84610b33f8546b515e0fd (127.28.112.1:39299), 79ece8ecd514451abbcebc8b74a37bf5 (127.28.112.3:44481)
I20260809 07:56:16.324172 20457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b7f2044be67427b81d023e28f131c7a" candidate_uuid: "f9609c8bb0354680b799dd060579a25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1d92bb8bd84610b33f8546b515e0fd"
I20260809 07:56:16.324234 20870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b7f2044be67427b81d023e28f131c7a" candidate_uuid: "f9609c8bb0354680b799dd060579a25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ece8ecd514451abbcebc8b74a37bf5"
I20260809 07:56:16.324285 20457 raft_consensus.cc:3037] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:56:16.324363 20870 raft_consensus.cc:3037] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:56:16.325294 20457 raft_consensus.cc:2454] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9609c8bb0354680b799dd060579a25a in term 1.
I20260809 07:56:16.325359 20870 raft_consensus.cc:2454] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9609c8bb0354680b799dd060579a25a in term 1.
I20260809 07:56:16.325592 20559 leader_election.cc:302] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d1d92bb8bd84610b33f8546b515e0fd, f9609c8bb0354680b799dd060579a25a; no voters:
I20260809 07:56:16.325793 20973 raft_consensus.cc:2781] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:56:16.326038 20973 raft_consensus.cc:686] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [term 1 LEADER]: Becoming Leader. State: Replica: f9609c8bb0354680b799dd060579a25a, State: Running, Role: LEADER
I20260809 07:56:16.326377 20973 consensus_queue.cc:237] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } } peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } }
I20260809 07:56:16.329563 20231 catalog_manager.cc:5495] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a reported cstate change: term changed from 0 to 1, leader changed from <none> to f9609c8bb0354680b799dd060579a25a (127.28.112.2). New cstate: current_term: 1 leader_uuid: "f9609c8bb0354680b799dd060579a25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } health_report { overall_health: UNKNOWN } } }
I20260809 07:56:16.341615 20956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:16.338663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.28.112.2:46401)
0809 07:56:16.338765 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:16.338768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:16.338780 (+ 12us) server_negotiation.cc:407] Connection header received
0809 07:56:16.338907 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:16.338910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:16.338960 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:16.339044 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:56:16.339228 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.339704 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.340475 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:16.340678 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:16.340741 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:16.340951 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:56:16.341346 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:56:16.341431 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:16.341472 (+ 41us) server_negotiation.cc:299] Negotiation successful
0809 07:56:16.341516 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 07:56:16.347116 20457 raft_consensus.cc:1261] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Refusing update from remote peer f9609c8bb0354680b799dd060579a25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:56:16.347116 20870 raft_consensus.cc:1261] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5 [term 1 FOLLOWER]: Refusing update from remote peer f9609c8bb0354680b799dd060579a25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:56:16.347810 20980 consensus_queue.cc:1035] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:56:16.347978 20973 consensus_queue.cc:1035] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 07:56:16.352715 20939 tablet.cc:2369] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:56:16.357437 20987 mvcc.cc:205] Tried to move back new op lower bound from 7316529874311041024 to 7316529874235404288. Current Snapshot: MvccSnapshot[applied={T|T < 7316529874311041024 or (T in {7316529874311041024})}]
I20260809 07:56:16.357546 20986 mvcc.cc:205] Tried to move back new op lower bound from 7316529874311041024 to 7316529874235404288. Current Snapshot: MvccSnapshot[applied={T|T < 7316529874311041024}]
I20260809 07:56:16.364249 20235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:56:16.364419 20235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:56:16.366782 20235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848: Sending DeleteTablet for 3 replicas of tablet 6b7f2044be67427b81d023e28f131c7a
I20260809 07:56:16.367517 20630 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b7f2044be67427b81d023e28f131c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:56:16 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:60980
I20260809 07:56:16.367597 20843 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b7f2044be67427b81d023e28f131c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:56:16 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:54568
I20260809 07:56:16.368003 21003 tablet_replica.cc:330] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a: stopping tablet replica
I20260809 07:56:16.368183 20428 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b7f2044be67427b81d023e28f131c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:56:16 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:48402
I20260809 07:56:16.368268 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 20333
I20260809 07:56:16.368273 21003 raft_consensus.cc:2229] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:56:16.368613 21003 raft_consensus.cc:2258] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:56:16.375116 21003 ts_tablet_manager.cc:1905] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:56:16.377938 21003 ts_tablet_manager.cc:1918] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:56:16.378036 21003 log.cc:1196] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/6b7f2044be67427b81d023e28f131c7a
I20260809 07:56:16.378283 21003 ts_tablet_manager.cc:1939] T 6b7f2044be67427b81d023e28f131c7a P f9609c8bb0354680b799dd060579a25a: Deleting consensus metadata
I20260809 07:56:16.379585 20216 catalog_manager.cc:4841] TS f9609c8bb0354680b799dd060579a25a (127.28.112.2:46401): tablet 6b7f2044be67427b81d023e28f131c7a (table test-table [id=9f281f3ce6d94c2d9a409de3ec4a9194]) successfully deleted
I20260809 07:56:16.379690 21005 tablet_replica.cc:330] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5: stopping tablet replica
I20260809 07:56:16.380008 21005 raft_consensus.cc:2229] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:56:16.380262 21005 raft_consensus.cc:2258] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:56:16.383136 21005 ts_tablet_manager.cc:1905] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:56:16.385198 21005 ts_tablet_manager.cc:1918] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:56:16.385284 21005 log.cc:1196] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/6b7f2044be67427b81d023e28f131c7a
I20260809 07:56:16.385584 21005 ts_tablet_manager.cc:1939] T 6b7f2044be67427b81d023e28f131c7a P 79ece8ecd514451abbcebc8b74a37bf5: Deleting consensus metadata
I20260809 07:56:16.386806 20214 catalog_manager.cc:4841] TS 79ece8ecd514451abbcebc8b74a37bf5 (127.28.112.3:44481): tablet 6b7f2044be67427b81d023e28f131c7a (table test-table [id=9f281f3ce6d94c2d9a409de3ec4a9194]) successfully deleted
W20260809 07:56:16.390350 20213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.112.1:39299 (error 108)
I20260809 07:56:16.390647 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 20531
I20260809 07:56:16.392899 20957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:16.392578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:39299 (local address 127.0.0.1:48408)
0809 07:56:16.392730 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:16.392808 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:39299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 07:56:16.393074 20213 catalog_manager.cc:4584] TS 5d1d92bb8bd84610b33f8546b515e0fd (127.28.112.1:39299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b7f2044be67427b81d023e28f131c7a: Network error: Client connection negotiation failed: client connection to 127.28.112.1:39299: connect: Connection refused (error 111)
I20260809 07:56:16.399325 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 20741
I20260809 07:56:16.412797 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 20190
I20260809 07:56:36.422067 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:36865
--webserver_interface=127.28.112.62
--webserver_port=40361
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:36865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:56:36.533685 21007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:56:36.533926 21007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:56:36.534000 21007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:56:36.537858 21007 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 07:56:36.537907 21007 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 07:56:36.537930 21007 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 07:56:36.537966 21007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:56:36.537992 21007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:56:36.538025 21007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:56:36.538049 21007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:56:36.538079 21007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:56:36.543195 21007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:36865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:36865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=40361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:56:36.544183 21007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:56:36.545168 21007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:56:36.558584 21017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:56:36.558626 21018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:56:36.558691 21007 server_base.cc:1034] running on GCE node
W20260809 07:56:36.558626 21020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:56:36.559317 21007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:56:36.560443 21007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:56:36.561604 21007 hybrid_clock.cc:648] HybridClock initialized: now 1786262196561580 us; error 31 us; skew 500 ppm
Aug 09 07:56:36 dist-test-slave-vpvm krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:56:36.564347 21007 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:56:36.565203 21007 webserver.cc:460] Webserver started at http://127.28.112.62:40361/ using document root <none> and password file <none>
I20260809 07:56:36.565724 21007 fs_manager.cc:359] Metadata directory not provided
I20260809 07:56:36.565794 21007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:56:36.569550 21007 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260809 07:56:36.571640 21028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:36.572810 21007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:56:36.572926 21007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "1739d096f8894e84ad0fd97c64277848"
format_stamp: "Formatted at 2026-08-09 07:56:15 on dist-test-slave-vpvm"
server_key: "7cc3dce32ca20324dbd7be418679d6f9"
server_key_iv: "34b70f224d5ea96ad18e2ed3f8bb869d"
server_key_version: "encryptionkey@0"
I20260809 07:56:36.573498 21007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:56:36.592952 21007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:56:36.595534 21007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:56:36.595706 21007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:56:36.602471 21007 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:36865
I20260809 07:56:36.602737 21104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:36865 every 8 connection(s)
I20260809 07:56:36.603544 21007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:56:36.607947 21105 sys_catalog.cc:263] Verifying existing consensus state
I20260809 07:56:36.609165 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 21007
I20260809 07:56:36.609560 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:39299
--local_ip_for_outbound_sockets=127.28.112.1
--tserver_master_addrs=127.28.112.62:36865
--webserver_port=34849
--webserver_interface=127.28.112.1
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 07:56:36.609665 21105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848: Bootstrap starting.
I20260809 07:56:36.626816 21105 log.cc:824] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848: Log is configured to *not* fsync() on all Append() calls
I20260809 07:56:36.635484 21105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 07:56:36.635799 21105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848: Bootstrap complete.
I20260809 07:56:36.639595 21105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1739d096f8894e84ad0fd97c64277848" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 36865 } }
I20260809 07:56:36.639981 21105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1739d096f8894e84ad0fd97c64277848, State: Initialized, Role: FOLLOWER
I20260809 07:56:36.640446 21105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1739d096f8894e84ad0fd97c64277848" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 36865 } }
I20260809 07:56:36.640571 21105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:56:36.640645 21105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:56:36.640739 21105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 1 FOLLOWER]: Advancing to term 2
I20260809 07:56:36.642545 21105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1739d096f8894e84ad0fd97c64277848" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 36865 } }
I20260809 07:56:36.642851 21105 leader_election.cc:302] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1739d096f8894e84ad0fd97c64277848; no voters:
I20260809 07:56:36.643158 21105 leader_election.cc:288] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 07:56:36.643283 21109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 07:56:36.643532 21109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 2 LEADER]: Becoming Leader. State: Replica: 1739d096f8894e84ad0fd97c64277848, State: Running, Role: LEADER
I20260809 07:56:36.643828 21109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1739d096f8894e84ad0fd97c64277848" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 36865 } }
I20260809 07:56:36.644315 21105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:56:36.645349 21111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1739d096f8894e84ad0fd97c64277848. Latest consensus state: current_term: 2 leader_uuid: "1739d096f8894e84ad0fd97c64277848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1739d096f8894e84ad0fd97c64277848" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 36865 } } }
I20260809 07:56:36.645385 21110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1739d096f8894e84ad0fd97c64277848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1739d096f8894e84ad0fd97c64277848" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 36865 } } }
I20260809 07:56:36.645478 21111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [sys.catalog]: This master's current role is: LEADER
I20260809 07:56:36.645484 21110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [sys.catalog]: This master's current role is: LEADER
I20260809 07:56:36.645820 21131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:56:36.650529 21131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:56:36.651362 21131 catalog_manager.cc:1259] Loaded cluster ID: 2ffe21266a6d42b89bac0f6cd98c05a2
I20260809 07:56:36.651422 21131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:56:36.653795 21131 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:56:36.654501 21131 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848: Loaded TSK: 0
I20260809 07:56:36.666195 21131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848: Generated new TSK 1
I20260809 07:56:36.666431 21131 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 07:56:36.726800 21107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:56:36.727027 21107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:56:36.727067 21107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:56:36.731005 21107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:56:36.731053 21107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:56:36.731112 21107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:56:36.736232 21107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:39299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=34849
--tserver_master_addrs=127.28.112.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:56:36.737157 21107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:56:36.738138 21107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:56:36.748332 21147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:56:36.748322 21149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:56:36.748323 21146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:56:36.748778 21107 server_base.cc:1034] running on GCE node
I20260809 07:56:36.749167 21107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:56:36.749769 21107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:56:36.750927 21107 hybrid_clock.cc:648] HybridClock initialized: now 1786262196750894 us; error 54 us; skew 500 ppm
Aug 09 07:56:36 dist-test-slave-vpvm krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:56:36.753533 21107 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:56:36.754421 21107 webserver.cc:460] Webserver started at http://127.28.112.1:34849/ using document root <none> and password file <none>
I20260809 07:56:36.755089 21107 fs_manager.cc:359] Metadata directory not provided
I20260809 07:56:36.755173 21107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:56:36.758881 21107 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260809 07:56:36.760883 21159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:36.762063 21107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:56:36.762162 21107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "5d1d92bb8bd84610b33f8546b515e0fd"
format_stamp: "Formatted at 2026-08-09 07:56:15 on dist-test-slave-vpvm"
server_key: "ca11be9f18842afb6a50b0f156aea67d"
server_key_iv: "244f05d4e9b2226a6a8edfaf93d7d69a"
server_key_version: "encryptionkey@0"
I20260809 07:56:36.762667 21107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:56:36.774856 21107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:56:36.777683 21107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:56:36.777843 21107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:56:36.778420 21107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:56:36.779512 21170 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 07:56:36.783420 21107 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 07:56:36.783475 21107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260809 07:56:36.783524 21107 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 07:56:36.785089 21107 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 07:56:36.785147 21107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260809 07:56:36.785435 21170 tablet_bootstrap.cc:492] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd: Bootstrap starting.
I20260809 07:56:36.795523 21107 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:39299
I20260809 07:56:36.795564 21338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:39299 every 8 connection(s)
I20260809 07:56:36.796706 21107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:56:36.804994 21170 log.cc:824] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd: Log is configured to *not* fsync() on all Append() calls
I20260809 07:56:36.806493 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 21107
I20260809 07:56:36.806980 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:46401
--local_ip_for_outbound_sockets=127.28.112.2
--tserver_master_addrs=127.28.112.62:36865
--webserver_port=34437
--webserver_interface=127.28.112.2
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 07:56:36 dist-test-slave-vpvm krb5kdc[20172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:56:36.813397 21170 tablet_bootstrap.cc:492] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 07:56:36.813680 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:36.798840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:34187 (local address 127.28.112.62:36865)
0809 07:56:36.799226 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:36.799247 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:36.799563 (+ 316us) server_negotiation.cc:407] Connection header received
0809 07:56:36.800638 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:36.800648 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:36.800938 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:36.801248 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:56:36.802419 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:36.803402 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:36.804550 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:36.805009 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:36.805171 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:36.807968 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:56:36.807975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:56:36.807992 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:56:36.808027 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:56:36.810058 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:36.810713 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:36.810719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:36.810725 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:36.810878 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:36.811274 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:36.811278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:36.811280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:36.811738 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:56:36.811897 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:36.812520 (+ 623us) server_negotiation.cc:299] Negotiation successful
0809 07:56:36.812838 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":131,"threads_started":1}
I20260809 07:56:36.813685 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:36.798859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:34187)
0809 07:56:36.799392 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:36.799440 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:36.800223 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:36.801323 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:36.801331 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:36.801737 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:56:36.802231 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:36.802246 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:36.803571 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:36.803574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:36.804419 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:36.804430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:36.805196 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:36.805201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:36.805383 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:36.805990 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:56:36.806013 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:56:36.807729 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:56:36.810182 (+ 2453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:36.810189 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:36.810201 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:36.810585 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:36.810979 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:36.810982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:36.810984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:36.811161 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:36.811874 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:56:36.811881 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:56:36.812149 (+ 268us) client_negotiation.cc:769] Sending connection context
0809 07:56:36.812346 (+ 197us) client_negotiation.cc:240] Negotiation successful
0809 07:56:36.812666 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":150,"threads_started":1}
I20260809 07:56:36.813848 21170 tablet_bootstrap.cc:492] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd: Bootstrap complete.
I20260809 07:56:36.815171 21170 ts_tablet_manager.cc:1397] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd: Time spent bootstrapping tablet: real 0.030s user 0.016s sys 0.012s
I20260809 07:56:36.816691 21341 heartbeater.cc:347] Connected to a master server at 127.28.112.62:36865
I20260809 07:56:36.816947 21341 heartbeater.cc:464] Registering TS with master...
I20260809 07:56:36.817517 21341 heartbeater.cc:511] Master 127.28.112.62:36865 requested a full tablet report, sending...
I20260809 07:56:36.817770 21170 raft_consensus.cc:348] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } } peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } }
I20260809 07:56:36.818217 21170 raft_consensus.cc:729] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1d92bb8bd84610b33f8546b515e0fd, State: Initialized, Role: FOLLOWER
I20260809 07:56:36.818638 21170 consensus_queue.cc:260] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } } peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } }
I20260809 07:56:36.819507 21170 ts_tablet_manager.cc:1428] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260809 07:56:36.821663 21053 ts_manager.cc:194] Registered new tserver with Master: 5d1d92bb8bd84610b33f8546b515e0fd (127.28.112.1:39299)
I20260809 07:56:36.824922 21053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:34187
I20260809 07:56:36.827489 21341 heartbeater.cc:503] Master 127.28.112.62:36865 was elected leader, sending a full tablet report...
Aug 09 07:56:36 dist-test-slave-vpvm krb5kdc[20172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for kudu/127.28.112.1@KRBTEST.COM
I20260809 07:56:36.834116 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:36.824342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:39299 (local address 127.0.0.1:32950)
0809 07:56:36.825051 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:36.825102 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:36.825259 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:36.825900 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:36.825906 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:36.826265 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:56:36.826361 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:36.826367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:36.827278 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:36.827281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:36.827948 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:36.827959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:36.828754 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:36.828757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:36.828857 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:36.829340 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:56:36.829353 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:56:36.830802 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:56:36.832431 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:36.832441 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:36.832503 (+ 62us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:36.832759 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:36.833029 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:36.833033 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:36.833035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:36.833114 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:36.833515 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:56:36.833521 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:56:36.833636 (+ 115us) client_negotiation.cc:769] Sending connection context
0809 07:56:36.833750 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 07:56:36.833923 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":120,"threads_started":1}
I20260809 07:56:36.834188 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:36.824399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32950 (local address 127.28.112.1:39299)
0809 07:56:36.824765 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:36.824776 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:36.825201 (+ 425us) server_negotiation.cc:407] Connection header received
0809 07:56:36.825419 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:36.825425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:36.825616 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:36.825746 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:56:36.826475 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:36.827169 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:36.828056 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:36.828512 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:36.828631 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:36.830925 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:56:36.830934 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:56:36.830947 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:56:36.830970 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:56:36.832330 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:36.832865 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:36.832870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:36.832871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:36.832945 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:36.833192 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:36.833195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:36.833197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:36.833406 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:56:36.833521 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:36.833842 (+ 321us) server_negotiation.cc:299] Negotiation successful
0809 07:56:36.834043 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":104,"threads_started":1}
I20260809 07:56:36.835383 21235 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b7f2044be67427b81d023e28f131c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:56:16 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:32950
I20260809 07:56:36.836103 21352 tablet_replica.cc:330] stopping tablet replica
I20260809 07:56:36.836359 21352 raft_consensus.cc:2229] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:56:36.836575 21352 raft_consensus.cc:2258] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:56:36.837445 21352 ts_tablet_manager.cc:1905] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:56:36.841009 21352 ts_tablet_manager.cc:1918] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:56:36.841096 21352 log.cc:1196] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/wal/wals/6b7f2044be67427b81d023e28f131c7a
I20260809 07:56:36.841321 21352 ts_tablet_manager.cc:1939] T 6b7f2044be67427b81d023e28f131c7a P 5d1d92bb8bd84610b33f8546b515e0fd: Deleting consensus metadata
I20260809 07:56:36.842890 21032 catalog_manager.cc:4841] TS 5d1d92bb8bd84610b33f8546b515e0fd (127.28.112.1:39299): tablet 6b7f2044be67427b81d023e28f131c7a (table test-table [id=9f281f3ce6d94c2d9a409de3ec4a9194]) successfully deleted
W20260809 07:56:36.921314 21346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:56:36.921550 21346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:56:36.921605 21346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:56:36.925550 21346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:56:36.925602 21346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:56:36.925681 21346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:56:36.930650 21346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:46401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=34437
--tserver_master_addrs=127.28.112.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:56:36.931608 21346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:56:36.932566 21346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:56:36.941625 21362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:56:36.941619 21361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:56:36.941619 21364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:56:36.941799 21346 server_base.cc:1034] running on GCE node
I20260809 07:56:36.942278 21346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:56:36.942850 21346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:56:36.944012 21346 hybrid_clock.cc:648] HybridClock initialized: now 1786262196943992 us; error 28 us; skew 500 ppm
Aug 09 07:56:36 dist-test-slave-vpvm krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:56:36.946704 21346 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:56:36.947479 21346 webserver.cc:460] Webserver started at http://127.28.112.2:34437/ using document root <none> and password file <none>
I20260809 07:56:36.947973 21346 fs_manager.cc:359] Metadata directory not provided
I20260809 07:56:36.948038 21346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:56:36.951614 21346 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260809 07:56:36.953467 21375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:36.954654 21346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:56:36.954758 21346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "f9609c8bb0354680b799dd060579a25a"
format_stamp: "Formatted at 2026-08-09 07:56:15 on dist-test-slave-vpvm"
server_key: "6e0c8c8a004a15ab67bc208e93e20604"
server_key_iv: "0f24c2c132a926f32ef99e99800aa770"
server_key_version: "encryptionkey@0"
I20260809 07:56:36.955297 21346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:56:36.973203 21346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:56:36.975878 21346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:56:36.976027 21346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:56:36.976584 21346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:56:36.977331 21346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:56:36.977377 21346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:36.977422 21346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:56:36.977444 21346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:36.985937 21346 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:46401
I20260809 07:56:36.985977 21559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:46401 every 8 connection(s)
I20260809 07:56:36.986923 21346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:56:36.994686 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 21346
I20260809 07:56:36.995134 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:44481
--local_ip_for_outbound_sockets=127.28.112.3
--tserver_master_addrs=127.28.112.62:36865
--webserver_port=42599
--webserver_interface=127.28.112.3
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 07:56:36 dist-test-slave-vpvm krb5kdc[20172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:56:36.999756 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:36.988601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:43435 (local address 127.28.112.62:36865)
0809 07:56:36.988733 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:36.988737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:36.989582 (+ 845us) server_negotiation.cc:407] Connection header received
0809 07:56:36.990847 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:36.990850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:36.990902 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:36.990973 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:56:36.992039 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:36.992565 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:36.994015 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:36.994245 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:36.994322 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:36.996594 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:56:36.996598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:56:36.996600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:56:36.996626 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:56:36.997807 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:36.998313 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:36.998316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:36.998318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:36.998398 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:36.998719 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:36.998723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:36.998725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:36.998900 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:56:36.998984 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:36.999462 (+ 478us) server_negotiation.cc:299] Negotiation successful
0809 07:56:36.999595 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 07:56:37.000474 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:36.988941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:43435)
0809 07:56:36.989425 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:36.989466 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:36.990655 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:36.991121 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:36.991127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:36.991456 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:56:36.991926 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:36.991939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:36.992717 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:36.992721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:36.993877 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:36.993888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:36.994359 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:36.994361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:36.994521 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:36.995048 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:56:36.995064 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:56:36.996433 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:56:36.997914 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:36.997920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:36.997935 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:36.998218 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:36.998481 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:36.998484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:36.998486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:36.998614 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:36.998997 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:56:36.999003 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:56:36.999230 (+ 227us) client_negotiation.cc:769] Sending connection context
0809 07:56:36.999383 (+ 153us) client_negotiation.cc:240] Negotiation successful
0809 07:56:36.999691 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":140,"threads_started":1}
I20260809 07:56:37.001724 21560 heartbeater.cc:347] Connected to a master server at 127.28.112.62:36865
I20260809 07:56:37.001973 21560 heartbeater.cc:464] Registering TS with master...
I20260809 07:56:37.002442 21560 heartbeater.cc:511] Master 127.28.112.62:36865 requested a full tablet report, sending...
I20260809 07:56:37.003566 21053 ts_manager.cc:194] Registered new tserver with Master: f9609c8bb0354680b799dd060579a25a (127.28.112.2:46401)
I20260809 07:56:37.004336 21053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:43435
W20260809 07:56:37.106115 21563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:56:37.106348 21563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:56:37.106388 21563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:56:37.110224 21563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:56:37.110272 21563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:56:37.110325 21563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:56:37.115286 21563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:44481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=42599
--tserver_master_addrs=127.28.112.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:56:37.116173 21563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:56:37.117080 21563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:56:37.126125 21574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:56:37.126169 21577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:56:37.126173 21579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:56:37.126302 21563 server_base.cc:1034] running on GCE node
I20260809 07:56:37.126752 21563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:56:37.127328 21563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:56:37.128484 21563 hybrid_clock.cc:648] HybridClock initialized: now 1786262197128465 us; error 29 us; skew 500 ppm
Aug 09 07:56:37 dist-test-slave-vpvm krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262197, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:56:37.131062 21563 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:56:37.131907 21563 webserver.cc:460] Webserver started at http://127.28.112.3:42599/ using document root <none> and password file <none>
I20260809 07:56:37.132426 21563 fs_manager.cc:359] Metadata directory not provided
I20260809 07:56:37.132503 21563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:56:37.136137 21563 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:56:37.138096 21589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:37.139248 21563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:56:37.139353 21563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "79ece8ecd514451abbcebc8b74a37bf5"
format_stamp: "Formatted at 2026-08-09 07:56:16 on dist-test-slave-vpvm"
server_key: "d75d76a517a5da229392abab36a4fbb4"
server_key_iv: "a72e5f8a5bdecffd53b208bd2614e8b3"
server_key_version: "encryptionkey@0"
I20260809 07:56:37.139837 21563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:56:37.166996 21563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:56:37.170027 21563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:56:37.170207 21563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:56:37.170794 21563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:56:37.171651 21563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:56:37.171720 21563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:37.171780 21563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:56:37.171803 21563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:37.180368 21563 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:44481
I20260809 07:56:37.180388 21764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:44481 every 8 connection(s)
I20260809 07:56:37.181389 21563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:56:37.182251 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 21563
I20260809 07:56:37.191192 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:37.183362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40542 (local address 127.28.112.62:36865)
0809 07:56:37.183491 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:37.183496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:37.183510 (+ 14us) server_negotiation.cc:407] Connection header received
0809 07:56:37.183556 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:37.183559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:37.183605 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:37.183674 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:56:37.184519 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.185088 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.186266 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.186489 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.186566 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:37.188331 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:56:37.188335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:56:37.188337 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:56:37.188359 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:56:37.189608 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:37.190076 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:37.190078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:37.190079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:37.190165 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:37.190418 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:37.190420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:37.190421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:37.190575 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:56:37.190711 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:37.190899 (+ 188us) server_negotiation.cc:299] Negotiation successful
0809 07:56:37.191013 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Aug 09 07:56:37 dist-test-slave-vpvm krb5kdc[20172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262197, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:56:37.196444 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:37.185020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:44545 (local address 127.28.112.62:36865)
0809 07:56:37.185237 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:37.185240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:37.185522 (+ 282us) server_negotiation.cc:407] Connection header received
0809 07:56:37.187124 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:37.187127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:37.187180 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:37.187260 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:56:37.188687 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.189184 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.190570 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.190888 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.190984 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:37.193397 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:56:37.193401 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:56:37.193403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:56:37.193428 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:56:37.194552 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:37.195016 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:37.195020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:37.195022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:37.195099 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:37.195409 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:37.195412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:37.195414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:37.195584 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:56:37.195682 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:37.196157 (+ 475us) server_negotiation.cc:299] Negotiation successful
0809 07:56:37.196277 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":74,"threads_started":1}
I20260809 07:56:37.197222 21769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:37.184035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:44545)
0809 07:56:37.185385 (+ 1350us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:37.185439 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:37.186921 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:37.187443 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:37.187453 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:37.187923 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:56:37.188545 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:37.188560 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.189300 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.189304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:37.190467 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:37.190477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.191039 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.191043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:37.191238 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:37.191743 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:56:37.191762 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:56:37.193245 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:56:37.194660 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:37.194666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:37.194680 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:37.194921 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:37.195183 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:37.195187 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:37.195189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:37.195320 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:37.195684 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:56:37.195691 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:56:37.195927 (+ 236us) client_negotiation.cc:769] Sending connection context
0809 07:56:37.196097 (+ 170us) client_negotiation.cc:240] Negotiation successful
0809 07:56:37.196422 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":144,"threads_started":1}
I20260809 07:56:37.198277 21765 heartbeater.cc:347] Connected to a master server at 127.28.112.62:36865
I20260809 07:56:37.198484 21765 heartbeater.cc:464] Registering TS with master...
I20260809 07:56:37.198941 21765 heartbeater.cc:511] Master 127.28.112.62:36865 requested a full tablet report, sending...
I20260809 07:56:37.199887 21053 ts_manager.cc:194] Registered new tserver with Master: 79ece8ecd514451abbcebc8b74a37bf5 (127.28.112.3:44481)
I20260809 07:56:37.200400 21053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:44545
I20260809 07:56:37.204492 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 07:56:37.208285 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:37.205075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40558 (local address 127.28.112.62:36865)
0809 07:56:37.205206 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:37.205210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:37.205368 (+ 158us) server_negotiation.cc:407] Connection header received
0809 07:56:37.205523 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:37.205526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:37.205621 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:37.205705 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:56:37.205919 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.206437 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.207255 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.207472 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.207558 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:37.207833 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:56:37.207950 (+ 117us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0809 07:56:37.208174 (+ 224us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40558: token expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260809 07:56:37.208349 21770 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40558: token expired
W20260809 07:56:37.208673 20941 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.112.62:36865)
I20260809 07:56:37.214690 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:37.208998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40562 (local address 127.28.112.62:36865)
0809 07:56:37.209105 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:37.209108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:37.209120 (+ 12us) server_negotiation.cc:407] Connection header received
0809 07:56:37.209205 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:37.209208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:37.209243 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:37.209300 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:56:37.209829 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.210320 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.211016 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.211210 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.211271 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:37.212365 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:56:37.212368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:56:37.212370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:56:37.212384 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:56:37.213473 (+ 1089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:37.213802 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:37.213804 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:37.213805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:37.213868 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:37.214085 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:37.214087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:37.214088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:37.214245 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:56:37.214342 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:37.214478 (+ 136us) server_negotiation.cc:299] Negotiation successful
0809 07:56:37.214562 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 07:56:37.216773 21053 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:56:37.219905 21053 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:56:37.227341 21235 tablet_service.cc:1467] Processing CreateTablet for tablet 91263c725ebc47bc8a37a2b75e06e75e (DEFAULT_TABLE table=test-table [id=70f523faf69d443f9ecc020fd7b8b911]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:56:37.228017 21235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91263c725ebc47bc8a37a2b75e06e75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:56:37.230917 21170 tablet_bootstrap.cc:492] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd: Bootstrap starting.
I20260809 07:56:37.231022 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:37.226293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:44481 (local address 127.0.0.1:39896)
0809 07:56:37.226394 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:37.226408 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:37.226485 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:37.227319 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:37.227322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:37.227339 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:56:37.227425 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:37.227431 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.228535 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.228539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:37.229498 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:37.229509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.230611 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.230615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:37.230710 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:37.230756 (+ 46us) client_negotiation.cc:769] Sending connection context
0809 07:56:37.230829 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 07:56:37.230908 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260809 07:56:37.231112 21780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:37.226466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39896 (local address 127.28.112.3:44481)
0809 07:56:37.226888 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:37.226899 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:37.226923 (+ 24us) server_negotiation.cc:407] Connection header received
0809 07:56:37.226999 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:37.227004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:37.227186 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:37.227326 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:56:37.227514 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.228434 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.229652 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.230473 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.230595 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:37.230755 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:37.230871 (+ 116us) server_negotiation.cc:299] Negotiation successful
0809 07:56:37.230969 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":113,"threads_started":1}
I20260809 07:56:37.231756 21170 tablet_bootstrap.cc:654] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd: Neither blocks nor log segments found. Creating new log.
I20260809 07:56:37.232244 21781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:37.226544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:46401 (local address 127.0.0.1:53766)
0809 07:56:37.226848 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:37.226861 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:37.226948 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:37.227739 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:37.227742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:37.227763 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:56:37.227844 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:37.227849 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.229070 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.229074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:37.230018 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:37.230027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.231830 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.231834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:37.231936 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:37.231961 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 07:56:37.232041 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 07:56:37.232110 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":90,"threads_started":1}
I20260809 07:56:37.232331 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:37.226839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53766 (local address 127.28.112.2:46401)
0809 07:56:37.227241 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:37.227251 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:37.227274 (+ 23us) server_negotiation.cc:407] Connection header received
0809 07:56:37.227348 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:37.227353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:37.227589 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:37.227752 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:56:37.227963 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.228953 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.230141 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.230721 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.231603 (+ 882us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:37.231740 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:37.232085 (+ 345us) server_negotiation.cc:299] Negotiation successful
0809 07:56:37.232193 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":116,"threads_started":1}
I20260809 07:56:37.233420 21170 tablet_bootstrap.cc:492] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd: No bootstrap required, opened a new log
I20260809 07:56:37.233516 21170 ts_tablet_manager.cc:1397] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260809 07:56:37.233940 21170 raft_consensus.cc:348] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } } peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } }
I20260809 07:56:37.234079 21170 raft_consensus.cc:374] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:56:37.234134 21170 raft_consensus.cc:729] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1d92bb8bd84610b33f8546b515e0fd, State: Initialized, Role: FOLLOWER
I20260809 07:56:37.234284 21170 consensus_queue.cc:260] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } } peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } }
I20260809 07:56:37.234191 21660 tablet_service.cc:1467] Processing CreateTablet for tablet 91263c725ebc47bc8a37a2b75e06e75e (DEFAULT_TABLE table=test-table [id=70f523faf69d443f9ecc020fd7b8b911]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:56:37.235165 21660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91263c725ebc47bc8a37a2b75e06e75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:56:37.235244 21170 ts_tablet_manager.cc:1428] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260809 07:56:37.235126 21459 tablet_service.cc:1467] Processing CreateTablet for tablet 91263c725ebc47bc8a37a2b75e06e75e (DEFAULT_TABLE table=test-table [id=70f523faf69d443f9ecc020fd7b8b911]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:56:37.236124 21459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91263c725ebc47bc8a37a2b75e06e75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:56:37.242300 21786 tablet_bootstrap.cc:492] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5: Bootstrap starting.
I20260809 07:56:37.242337 21787 tablet_bootstrap.cc:492] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a: Bootstrap starting.
I20260809 07:56:37.244585 21786 tablet_bootstrap.cc:654] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5: Neither blocks nor log segments found. Creating new log.
I20260809 07:56:37.244731 21787 tablet_bootstrap.cc:654] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a: Neither blocks nor log segments found. Creating new log.
I20260809 07:56:37.245333 21786 log.cc:824] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5: Log is configured to *not* fsync() on all Append() calls
I20260809 07:56:37.245469 21787 log.cc:824] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a: Log is configured to *not* fsync() on all Append() calls
I20260809 07:56:37.247486 21787 tablet_bootstrap.cc:492] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a: No bootstrap required, opened a new log
I20260809 07:56:37.247488 21786 tablet_bootstrap.cc:492] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5: No bootstrap required, opened a new log
I20260809 07:56:37.247694 21787 ts_tablet_manager.cc:1397] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 07:56:37.247694 21786 ts_tablet_manager.cc:1397] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 07:56:37.251564 21786 raft_consensus.cc:348] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } } peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } }
I20260809 07:56:37.251564 21787 raft_consensus.cc:348] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } } peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } }
I20260809 07:56:37.251850 21787 raft_consensus.cc:374] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:56:37.251850 21786 raft_consensus.cc:374] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:56:37.251911 21786 raft_consensus.cc:729] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ece8ecd514451abbcebc8b74a37bf5, State: Initialized, Role: FOLLOWER
I20260809 07:56:37.251911 21787 raft_consensus.cc:729] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9609c8bb0354680b799dd060579a25a, State: Initialized, Role: FOLLOWER
I20260809 07:56:37.252377 21786 consensus_queue.cc:260] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } } peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } }
I20260809 07:56:37.252377 21787 consensus_queue.cc:260] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } } peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } }
I20260809 07:56:37.253037 21765 heartbeater.cc:503] Master 127.28.112.62:36865 was elected leader, sending a full tablet report...
I20260809 07:56:37.253037 21560 heartbeater.cc:503] Master 127.28.112.62:36865 was elected leader, sending a full tablet report...
I20260809 07:56:37.253399 21787 ts_tablet_manager.cc:1428] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 07:56:37.253399 21786 ts_tablet_manager.cc:1428] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260809 07:56:37.297897 21342 tablet.cc:2369] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:56:37.318755 21791 raft_consensus.cc:482] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:56:37.318956 21791 raft_consensus.cc:504] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } } peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } }
I20260809 07:56:37.320116 21791 leader_election.cc:288] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79ece8ecd514451abbcebc8b74a37bf5 (127.28.112.3:44481), 5d1d92bb8bd84610b33f8546b515e0fd (127.28.112.1:39299)
I20260809 07:56:37.323856 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:37.320335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:44481 (local address 127.28.112.2:34097)
0809 07:56:37.320518 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:37.320533 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:37.320620 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:37.320871 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:37.320874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:37.320892 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:56:37.321013 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:37.321021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.321869 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.321873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:37.322750 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:37.322760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.323509 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.323513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:37.323618 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:37.323641 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 07:56:37.323703 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 07:56:37.323755 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":1}
I20260809 07:56:37.323920 21780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:37.320511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:34097 (local address 127.28.112.3:44481)
0809 07:56:37.320639 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:37.320643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:37.320660 (+ 17us) server_negotiation.cc:407] Connection header received
0809 07:56:37.320715 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:37.320718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:37.320765 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:37.320861 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:56:37.321133 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.321764 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.322898 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.323390 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.323511 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:37.323562 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:37.323734 (+ 172us) server_negotiation.cc:299] Negotiation successful
0809 07:56:37.323802 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 07:56:37.324502 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:37.320573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:38919 (local address 127.28.112.1:39299)
0809 07:56:37.320690 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:37.320695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:37.320846 (+ 151us) server_negotiation.cc:407] Connection header received
0809 07:56:37.320961 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:37.320964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:37.321026 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:37.321119 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:56:37.321357 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.322009 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.323226 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.323856 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.323967 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:37.324110 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:37.324337 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 07:56:37.324402 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 07:56:37.324498 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:37.320379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:39299 (local address 127.28.112.2:38919)
0809 07:56:37.320768 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:37.320781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:37.320870 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:37.321133 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:37.321136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:37.321154 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:56:37.321262 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:37.321267 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.322117 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.322120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:37.323087 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:37.323097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.324113 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.324118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:37.324214 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:37.324233 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 07:56:37.324306 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 07:56:37.324372 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":126,"threads_started":1}
I20260809 07:56:37.324581 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91263c725ebc47bc8a37a2b75e06e75e" candidate_uuid: "f9609c8bb0354680b799dd060579a25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ece8ecd514451abbcebc8b74a37bf5" is_pre_election: true
I20260809 07:56:37.324892 21686 raft_consensus.cc:2454] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9609c8bb0354680b799dd060579a25a in term 0.
I20260809 07:56:37.325024 21265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91263c725ebc47bc8a37a2b75e06e75e" candidate_uuid: "f9609c8bb0354680b799dd060579a25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" is_pre_election: true
I20260809 07:56:37.325295 21265 raft_consensus.cc:2454] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9609c8bb0354680b799dd060579a25a in term 0.
I20260809 07:56:37.325371 21378 leader_election.cc:302] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ece8ecd514451abbcebc8b74a37bf5, f9609c8bb0354680b799dd060579a25a; no voters:
I20260809 07:56:37.325645 21791 raft_consensus.cc:2781] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:56:37.325721 21791 raft_consensus.cc:482] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:56:37.325750 21791 raft_consensus.cc:3037] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:56:37.326784 21791 raft_consensus.cc:504] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } } peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } }
I20260809 07:56:37.327092 21791 leader_election.cc:288] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [CANDIDATE]: Term 1 election: Requested vote from peers 79ece8ecd514451abbcebc8b74a37bf5 (127.28.112.3:44481), 5d1d92bb8bd84610b33f8546b515e0fd (127.28.112.1:39299)
I20260809 07:56:37.327427 21265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91263c725ebc47bc8a37a2b75e06e75e" candidate_uuid: "f9609c8bb0354680b799dd060579a25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1d92bb8bd84610b33f8546b515e0fd"
I20260809 07:56:37.327531 21686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91263c725ebc47bc8a37a2b75e06e75e" candidate_uuid: "f9609c8bb0354680b799dd060579a25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ece8ecd514451abbcebc8b74a37bf5"
I20260809 07:56:37.327553 21265 raft_consensus.cc:3037] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:56:37.327662 21686 raft_consensus.cc:3037] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:56:37.328629 21265 raft_consensus.cc:2454] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9609c8bb0354680b799dd060579a25a in term 1.
I20260809 07:56:37.328794 21686 raft_consensus.cc:2454] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9609c8bb0354680b799dd060579a25a in term 1.
I20260809 07:56:37.328883 21377 leader_election.cc:302] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d1d92bb8bd84610b33f8546b515e0fd, f9609c8bb0354680b799dd060579a25a; no voters:
I20260809 07:56:37.329035 21791 raft_consensus.cc:2781] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:56:37.329257 21791 raft_consensus.cc:686] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [term 1 LEADER]: Becoming Leader. State: Replica: f9609c8bb0354680b799dd060579a25a, State: Running, Role: LEADER
I20260809 07:56:37.329574 21791 consensus_queue.cc:237] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } } peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } }
I20260809 07:56:37.332587 21051 catalog_manager.cc:5495] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a reported cstate change: term changed from 0 to 1, leader changed from <none> to f9609c8bb0354680b799dd060579a25a (127.28.112.2). New cstate: current_term: 1 leader_uuid: "f9609c8bb0354680b799dd060579a25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } health_report { overall_health: UNKNOWN } } }
I20260809 07:56:37.360001 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:37.356382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53776 (local address 127.28.112.2:46401)
0809 07:56:37.356504 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:37.356508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:37.356523 (+ 15us) server_negotiation.cc:407] Connection header received
0809 07:56:37.356700 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:37.356703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:37.356774 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:37.356866 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:56:37.357120 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.357659 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.358569 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:37.358810 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:37.358898 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:37.359201 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:56:37.359631 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:56:37.359733 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:37.359805 (+ 72us) server_negotiation.cc:299] Negotiation successful
0809 07:56:37.359858 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 07:56:37.365566 21265 raft_consensus.cc:1261] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Refusing update from remote peer f9609c8bb0354680b799dd060579a25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:56:37.365562 21686 raft_consensus.cc:1261] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5 [term 1 FOLLOWER]: Refusing update from remote peer f9609c8bb0354680b799dd060579a25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:56:37.366205 21791 consensus_queue.cc:1035] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:56:37.366386 21794 consensus_queue.cc:1035] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:56:37.371671 21800 mvcc.cc:205] Tried to move back new op lower bound from 7316529960402563072 to 7316529960264695808. Current Snapshot: MvccSnapshot[applied={T|T < 7316529960402563072}]
I20260809 07:56:37.373927 21799 mvcc.cc:205] Tried to move back new op lower bound from 7316529960402563072 to 7316529960264695808. Current Snapshot: MvccSnapshot[applied={T|T < 7316529960402563072 or (T in {7316529960402563072})}]
I20260809 07:56:37.379405 21051 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:56:37.379540 21051 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:56:37.381673 21051 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848: Sending DeleteTablet for 3 replicas of tablet 91263c725ebc47bc8a37a2b75e06e75e
I20260809 07:56:37.382356 21459 tablet_service.cc:1514] Processing DeleteTablet for tablet 91263c725ebc47bc8a37a2b75e06e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:56:37 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:53766
I20260809 07:56:37.382431 21235 tablet_service.cc:1514] Processing DeleteTablet for tablet 91263c725ebc47bc8a37a2b75e06e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:56:37 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:32950
I20260809 07:56:37.382429 21660 tablet_service.cc:1514] Processing DeleteTablet for tablet 91263c725ebc47bc8a37a2b75e06e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:56:37 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:39896
I20260809 07:56:37.382766 21813 tablet_replica.cc:330] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a: stopping tablet replica
I20260809 07:56:37.382952 21814 tablet_replica.cc:330] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5: stopping tablet replica
I20260809 07:56:37.383049 21813 raft_consensus.cc:2229] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:56:37.383082 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 21107
I20260809 07:56:37.415418 21813 raft_consensus.cc:2258] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:56:37.418167 21813 ts_tablet_manager.cc:1905] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:56:37.420675 21813 ts_tablet_manager.cc:1918] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:56:37.420781 21813 log.cc:1196] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/91263c725ebc47bc8a37a2b75e06e75e
I20260809 07:56:37.421121 21813 ts_tablet_manager.cc:1939] T 91263c725ebc47bc8a37a2b75e06e75e P f9609c8bb0354680b799dd060579a25a: Deleting consensus metadata
I20260809 07:56:37.422610 21814 raft_consensus.cc:2229] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:56:37.422757 21035 catalog_manager.cc:4841] TS f9609c8bb0354680b799dd060579a25a (127.28.112.2:46401): tablet 91263c725ebc47bc8a37a2b75e06e75e (table test-table [id=70f523faf69d443f9ecc020fd7b8b911]) successfully deleted
I20260809 07:56:37.422979 21814 raft_consensus.cc:2258] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:56:37.424330 21814 ts_tablet_manager.cc:1905] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 07:56:37.425815 21032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.112.1:39299 (error 108)
I20260809 07:56:37.426653 21814 ts_tablet_manager.cc:1918] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:56:37.426739 21814 log.cc:1196] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/91263c725ebc47bc8a37a2b75e06e75e
I20260809 07:56:37.427067 21814 ts_tablet_manager.cc:1939] T 91263c725ebc47bc8a37a2b75e06e75e P 79ece8ecd514451abbcebc8b74a37bf5: Deleting consensus metadata
W20260809 07:56:37.427147 21781 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:56:37.426827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:39299 (local address 127.0.0.1:32960)
0809 07:56:37.426963 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:37.427042 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:39299: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":39}
W20260809 07:56:37.427279 21032 catalog_manager.cc:4584] TS 5d1d92bb8bd84610b33f8546b515e0fd (127.28.112.1:39299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91263c725ebc47bc8a37a2b75e06e75e: Network error: Client connection negotiation failed: client connection to 127.28.112.1:39299: connect: Connection reset by peer (error 104)
I20260809 07:56:37.428146 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 21346
I20260809 07:56:37.428211 21033 catalog_manager.cc:4841] TS 79ece8ecd514451abbcebc8b74a37bf5 (127.28.112.3:44481): tablet 91263c725ebc47bc8a37a2b75e06e75e (table test-table [id=70f523faf69d443f9ecc020fd7b8b911]) successfully deleted
I20260809 07:56:37.454073 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 21563
I20260809 07:56:37.462119 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 21007
I20260809 07:56:57.476529 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:36865
--webserver_interface=127.28.112.62
--webserver_port=40361
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:36865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:56:57.590507 21817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:56:57.590744 21817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:56:57.590798 21817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:56:57.594645 21817 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 07:56:57.594688 21817 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 07:56:57.594708 21817 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 07:56:57.594726 21817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:56:57.594751 21817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:56:57.594784 21817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:56:57.594813 21817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:56:57.594834 21817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:56:57.599828 21817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:36865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:36865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=40361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:56:57.600782 21817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:56:57.601722 21817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:56:57.610045 21827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:56:57.610086 21826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:56:57.610057 21817 server_base.cc:1034] running on GCE node
W20260809 07:56:57.610088 21829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:56:57.611068 21817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:56:57.612121 21817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:56:57.613287 21817 hybrid_clock.cc:648] HybridClock initialized: now 1786262217613266 us; error 38 us; skew 500 ppm
Aug 09 07:56:57 dist-test-slave-vpvm krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262217, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:56:57.615893 21817 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:56:57.616689 21817 webserver.cc:460] Webserver started at http://127.28.112.62:40361/ using document root <none> and password file <none>
I20260809 07:56:57.617184 21817 fs_manager.cc:359] Metadata directory not provided
I20260809 07:56:57.617242 21817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:56:57.620891 21817 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:56:57.622951 21841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:57.624090 21817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:56:57.624210 21817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "1739d096f8894e84ad0fd97c64277848"
format_stamp: "Formatted at 2026-08-09 07:56:15 on dist-test-slave-vpvm"
server_key: "7cc3dce32ca20324dbd7be418679d6f9"
server_key_iv: "34b70f224d5ea96ad18e2ed3f8bb869d"
server_key_version: "encryptionkey@0"
I20260809 07:56:57.624688 21817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:56:57.639470 21817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:56:57.641887 21817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:56:57.642066 21817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:56:57.648494 21817 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:36865
I20260809 07:56:57.648731 21931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:36865 every 8 connection(s)
I20260809 07:56:57.649482 21817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:56:57.653581 21932 sys_catalog.cc:263] Verifying existing consensus state
I20260809 07:56:57.653677 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 21817
I20260809 07:56:57.654163 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:39299
--local_ip_for_outbound_sockets=127.28.112.1
--tserver_master_addrs=127.28.112.62:36865
--webserver_port=34849
--webserver_interface=127.28.112.1
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 07:56:57.655680 21932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848: Bootstrap starting.
I20260809 07:56:57.672480 21932 log.cc:824] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848: Log is configured to *not* fsync() on all Append() calls
I20260809 07:56:57.683512 21932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 07:56:57.683871 21932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848: Bootstrap complete.
I20260809 07:56:57.687608 21932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1739d096f8894e84ad0fd97c64277848" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 36865 } }
I20260809 07:56:57.687978 21932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1739d096f8894e84ad0fd97c64277848, State: Initialized, Role: FOLLOWER
I20260809 07:56:57.688441 21932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1739d096f8894e84ad0fd97c64277848" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 36865 } }
I20260809 07:56:57.688555 21932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:56:57.688616 21932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:56:57.688709 21932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 2 FOLLOWER]: Advancing to term 3
I20260809 07:56:57.690528 21932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1739d096f8894e84ad0fd97c64277848" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 36865 } }
I20260809 07:56:57.690842 21932 leader_election.cc:302] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1739d096f8894e84ad0fd97c64277848; no voters:
I20260809 07:56:57.691133 21932 leader_election.cc:288] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 07:56:57.691218 21937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 07:56:57.691449 21937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [term 3 LEADER]: Becoming Leader. State: Replica: 1739d096f8894e84ad0fd97c64277848, State: Running, Role: LEADER
I20260809 07:56:57.691820 21937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1739d096f8894e84ad0fd97c64277848" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 36865 } }
I20260809 07:56:57.692375 21932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:56:57.692761 21937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1739d096f8894e84ad0fd97c64277848. Latest consensus state: current_term: 3 leader_uuid: "1739d096f8894e84ad0fd97c64277848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1739d096f8894e84ad0fd97c64277848" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 36865 } } }
I20260809 07:56:57.692963 21937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [sys.catalog]: This master's current role is: LEADER
I20260809 07:56:57.693266 21939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1739d096f8894e84ad0fd97c64277848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1739d096f8894e84ad0fd97c64277848" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 36865 } } }
I20260809 07:56:57.693439 21939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848 [sys.catalog]: This master's current role is: LEADER
I20260809 07:56:57.693536 21949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:56:57.698699 21949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:56:57.699573 21949 catalog_manager.cc:1259] Loaded cluster ID: 2ffe21266a6d42b89bac0f6cd98c05a2
I20260809 07:56:57.699647 21949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:56:57.702142 21949 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:56:57.702907 21949 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848: Loaded TSK: 0
I20260809 07:56:57.702993 21949 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1739d096f8894e84ad0fd97c64277848: Loaded TSK: 1
F20260809 07:56:57.704218 21949 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2e1f4449ad google::LogMessage::Fail() at ??:0
@ 0x7f2e1f44788a google::LogMessage::SendToLog() at ??:0
@ 0x7f2e1f4444de google::LogMessage::Flush() at ??:0
@ 0x7f2e1f445a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2e2210818d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f2e22100f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f2e29dad7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f2e29d8799c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f2e29d88178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f2e29dc9df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f2e29de882b std::function<>::operator()() at ??:0
@ 0x7f2e29d87b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f2e29d88f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2e29d85579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f2e29dc9640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f2e216963d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2e21696c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f2e216983f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f2e21687512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2e22ccb6db start_thread at ??:0
@ 0x7f2e1df4261f clone at ??:0
W20260809 07:56:57.791273 21934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:56:57.791580 21934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:56:57.791641 21934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:56:57.797475 21934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:56:57.797545 21934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:56:57.797624 21934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:56:57.805137 21934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:39299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=34849
--tserver_master_addrs=127.28.112.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:56:57.806432 21934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:56:57.807734 21934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:56:57.818324 22004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:56:57.818331 22001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:56:57.818439 22002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:56:57.819191 21934 server_base.cc:1034] running on GCE node
I20260809 07:56:57.819620 21934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:56:57.820429 21934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:56:57.821627 21934 hybrid_clock.cc:648] HybridClock initialized: now 1786262217821607 us; error 37 us; skew 500 ppm
Aug 09 07:56:57 dist-test-slave-vpvm krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262217, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:56:57.824874 21934 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:56:57.825996 21934 webserver.cc:460] Webserver started at http://127.28.112.1:34849/ using document root <none> and password file <none>
I20260809 07:56:57.826699 21934 fs_manager.cc:359] Metadata directory not provided
I20260809 07:56:57.826795 21934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:56:57.831624 21934 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:56:57.834136 22014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:57.835569 21934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 07:56:57.835700 21934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "5d1d92bb8bd84610b33f8546b515e0fd"
format_stamp: "Formatted at 2026-08-09 07:56:15 on dist-test-slave-vpvm"
server_key: "ca11be9f18842afb6a50b0f156aea67d"
server_key_iv: "244f05d4e9b2226a6a8edfaf93d7d69a"
server_key_version: "encryptionkey@0"
I20260809 07:56:57.836385 21934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:56:57.860008 21934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:56:57.863627 21934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:56:57.863824 21934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:56:57.864549 21934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:56:57.865841 22026 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 07:56:57.870051 21934 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 07:56:57.870131 21934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260809 07:56:57.870209 21934 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 07:56:57.872397 21934 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 07:56:57.872457 21934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260809 07:56:57.872725 22026 tablet_bootstrap.cc:492] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd: Bootstrap starting.
I20260809 07:56:57.883059 21934 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:39299
I20260809 07:56:57.883127 22189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:39299 every 8 connection(s)
I20260809 07:56:57.884385 21934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:56:57.891629 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 21934
I20260809 07:56:57.891687 22026 log.cc:824] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd: Log is configured to *not* fsync() on all Append() calls
I20260809 07:56:57.892110 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:46401
--local_ip_for_outbound_sockets=127.28.112.2
--tserver_master_addrs=127.28.112.62:36865
--webserver_port=34437
--webserver_interface=127.28.112.2
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 07:56:57.899803 22026 tablet_bootstrap.cc:492] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 07:56:57.900214 22026 tablet_bootstrap.cc:492] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd: Bootstrap complete.
I20260809 07:56:57.901473 22026 ts_tablet_manager.cc:1397] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd: Time spent bootstrapping tablet: real 0.029s user 0.022s sys 0.003s
I20260809 07:56:57.904671 22026 raft_consensus.cc:348] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } } peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } }
I20260809 07:56:57.905197 22026 raft_consensus.cc:729] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1d92bb8bd84610b33f8546b515e0fd, State: Initialized, Role: FOLLOWER
I20260809 07:56:57.905727 22026 consensus_queue.cc:260] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } } peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } }
I20260809 07:56:57.906698 22026 ts_tablet_manager.cc:1428] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
W20260809 07:56:57.945837 22196 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:56:57.886736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:38571)
0809 07:56:57.887322 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:57.887379 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:57.888421 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:57.945104 (+ 56683us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":168,"threads_started":1}
W20260809 07:56:57.946386 22190 heartbeater.cc:644] Failed to heartbeat to 127.28.112.62:36865 (0 consecutive failures): Network error: Failed to ping master at 127.28.112.62:36865: Client connection negotiation failed: client connection to 127.28.112.62:36865: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 07:56:57.947149 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:57.946902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:54117)
0809 07:56:57.947030 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:57.947092 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 07:56:57.947795 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:57.947602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:34169)
0809 07:56:57.947703 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:57.947745 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 07:56:57.947952 22190 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 07:56:58.006570 22200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:56:58.006794 22200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:56:58.006851 22200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:56:58.010738 22200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:56:58.010797 22200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:56:58.010874 22200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:56:58.015853 22200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:46401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=34437
--tserver_master_addrs=127.28.112.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:56:58.016798 22200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:56:58.017750 22200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:56:58.026746 22216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:56:58.026782 22212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:56:58.026746 22210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:56:58.026971 22200 server_base.cc:1034] running on GCE node
I20260809 07:56:58.027462 22200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:56:58.028070 22200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:56:58.029224 22200 hybrid_clock.cc:648] HybridClock initialized: now 1786262218029210 us; error 30 us; skew 500 ppm
Aug 09 07:56:58 dist-test-slave-vpvm krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262218, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:56:58.031750 22200 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:56:58.032524 22200 webserver.cc:460] Webserver started at http://127.28.112.2:34437/ using document root <none> and password file <none>
I20260809 07:56:58.033034 22200 fs_manager.cc:359] Metadata directory not provided
I20260809 07:56:58.033100 22200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:56:58.036667 22200 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:56:58.038599 22229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:58.039705 22200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 07:56:58.039812 22200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "f9609c8bb0354680b799dd060579a25a"
format_stamp: "Formatted at 2026-08-09 07:56:15 on dist-test-slave-vpvm"
server_key: "6e0c8c8a004a15ab67bc208e93e20604"
server_key_iv: "0f24c2c132a926f32ef99e99800aa770"
server_key_version: "encryptionkey@0"
I20260809 07:56:58.040302 22200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:56:58.049794 22200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:56:58.052326 22200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:56:58.052489 22200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:56:58.053018 22200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:56:58.053785 22200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:56:58.053833 22200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:58.053884 22200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:56:58.053911 22200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:58.062126 22200 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:46401
I20260809 07:56:58.062178 22404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:46401 every 8 connection(s)
I20260809 07:56:58.063053 22200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:56:58.066731 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:58.064995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:35795)
0809 07:56:58.065430 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:58.065529 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":114,"threads_started":1}
W20260809 07:56:58.067142 22405 heartbeater.cc:644] Failed to heartbeat to 127.28.112.62:36865 (0 consecutive failures): Network error: Failed to ping master at 127.28.112.62:36865: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
I20260809 07:56:58.067711 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:58.067481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:37945)
0809 07:56:58.067606 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:58.067657 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 07:56:58.068279 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:58.068078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:40539)
0809 07:56:58.068186 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:58.068230 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 07:56:58.068435 22405 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 07:56:58.071774 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 22200
I20260809 07:56:58.072161 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:44481
--local_ip_for_outbound_sockets=127.28.112.3
--tserver_master_addrs=127.28.112.62:36865
--webserver_port=42599
--webserver_interface=127.28.112.3
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:56:58.181878 22408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:56:58.182130 22408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:56:58.182169 22408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:56:58.185990 22408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:56:58.186043 22408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:56:58.186096 22408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:56:58.191073 22408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:44481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=42599
--tserver_master_addrs=127.28.112.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:56:58.191982 22408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:56:58.192904 22408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:56:58.201814 22418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:56:58.201855 22422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:56:58.201851 22419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:56:58.201929 22408 server_base.cc:1034] running on GCE node
I20260809 07:56:58.202481 22408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:56:58.203053 22408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:56:58.204211 22408 hybrid_clock.cc:648] HybridClock initialized: now 1786262218204191 us; error 31 us; skew 500 ppm
Aug 09 07:56:58 dist-test-slave-vpvm krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262218, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:56:58.206741 22408 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:56:58.207562 22408 webserver.cc:460] Webserver started at http://127.28.112.3:42599/ using document root <none> and password file <none>
I20260809 07:56:58.208078 22408 fs_manager.cc:359] Metadata directory not provided
I20260809 07:56:58.208148 22408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:56:58.211804 22408 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:56:58.213776 22436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:58.214920 22408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 07:56:58.215024 22408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "79ece8ecd514451abbcebc8b74a37bf5"
format_stamp: "Formatted at 2026-08-09 07:56:16 on dist-test-slave-vpvm"
server_key: "d75d76a517a5da229392abab36a4fbb4"
server_key_iv: "a72e5f8a5bdecffd53b208bd2614e8b3"
server_key_version: "encryptionkey@0"
I20260809 07:56:58.215559 22408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:56:58.250453 22408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:56:58.254528 22408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:56:58.254804 22408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:56:58.255440 22408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:56:58.256395 22408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:56:58.256444 22408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:58.256498 22408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:56:58.256526 22408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:56:58.264973 22408 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:44481
I20260809 07:56:58.265025 22623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:44481 every 8 connection(s)
I20260809 07:56:58.265934 22408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:56:58.269104 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 22408
I20260809 07:56:58.269845 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:58.268033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:60547)
0809 07:56:58.268496 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:58.268601 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":120,"threads_started":1}
W20260809 07:56:58.270328 22624 heartbeater.cc:644] Failed to heartbeat to 127.28.112.62:36865 (0 consecutive failures): Network error: Failed to ping master at 127.28.112.62:36865: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
I20260809 07:56:58.270846 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:58.270663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:39793)
0809 07:56:58.270752 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:58.270795 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260809 07:56:58.271306 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:58.271149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:38611)
0809 07:56:58.271224 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:58.271261 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260809 07:56:58.271440 22624 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 07:56:58.948896 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:58.948487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:47989)
0809 07:56:58.948730 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:58.948785 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":82,"threads_started":1}
I20260809 07:56:59.069319 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:59.068939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:37249)
0809 07:56:59.069168 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:59.069217 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":78,"threads_started":1}
I20260809 07:56:59.272269 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:59.271895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:51669)
0809 07:56:59.272117 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:59.272169 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260809 07:56:59.366577 22634 raft_consensus.cc:482] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:56:59.366796 22634 raft_consensus.cc:504] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } } peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } }
I20260809 07:56:59.368028 22634 leader_election.cc:288] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79ece8ecd514451abbcebc8b74a37bf5 (127.28.112.3:44481), f9609c8bb0354680b799dd060579a25a (127.28.112.2:46401)
Aug 09 07:56:59 dist-test-slave-vpvm krb5kdc[20172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262217, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.2@KRBTEST.COM
Aug 09 07:56:59 dist-test-slave-vpvm krb5kdc[20172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262217, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.3@KRBTEST.COM
I20260809 07:56:59.381429 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:59.368269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:46401 (local address 127.28.112.1:45783)
0809 07:56:59.368403 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:59.368417 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:59.368530 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:59.370216 (+ 1686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:59.370221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:59.370748 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:56:59.371230 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:59.371243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:59.372470 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:59.372473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:59.373383 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:59.373392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:59.374301 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:59.374305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:59.374479 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:59.375132 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:56:59.375155 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:56:59.376732 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:56:59.379001 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:59.379006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:59.379017 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:59.379274 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:59.379623 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:59.379626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:59.379628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:59.379794 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:59.380466 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:56:59.380471 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:56:59.380699 (+ 228us) client_negotiation.cc:769] Sending connection context
0809 07:56:59.380904 (+ 205us) client_negotiation.cc:240] Negotiation successful
0809 07:56:59.381298 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":244,"spinlock_wait_cycles":13056}
I20260809 07:56:59.381477 22638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:59.368451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:45783 (local address 127.28.112.2:46401)
0809 07:56:59.368768 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:59.368791 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:59.368827 (+ 36us) server_negotiation.cc:407] Connection header received
0809 07:56:59.369493 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:59.369500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:59.369844 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:59.370161 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:56:59.371393 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:59.372338 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:59.373562 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:59.374175 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:59.374333 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:59.376965 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:56:59.376971 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:56:59.376981 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:56:59.377008 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:56:59.378880 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:59.379380 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:59.379386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:59.379391 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:59.379524 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:59.379880 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:59.379883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:59.379885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:59.380352 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:56:59.380470 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:59.381028 (+ 558us) server_negotiation.cc:299] Negotiation successful
0809 07:56:59.381295 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":96,"threads_started":1}
I20260809 07:56:59.381501 22642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:59.368493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:37235 (local address 127.28.112.3:44481)
0809 07:56:59.368887 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0809 07:56:59.368911 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0809 07:56:59.368952 (+ 41us) server_negotiation.cc:407] Connection header received
0809 07:56:59.369561 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:56:59.369576 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:56:59.369859 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:56:59.370153 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:56:59.371385 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:59.372336 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:59.373537 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:56:59.374139 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:56:59.374316 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:59.377252 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:56:59.377258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:56:59.377273 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:56:59.377305 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:56:59.379338 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:59.379735 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:59.379739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:59.379743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:59.379845 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:56:59.380093 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:56:59.380096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:56:59.380098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:56:59.380537 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:56:59.380622 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 07:56:59.381023 (+ 401us) server_negotiation.cc:299] Negotiation successful
0809 07:56:59.381295 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":76,"threads_started":1}
I20260809 07:56:59.381428 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:59.368269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:44481 (local address 127.28.112.1:37235)
0809 07:56:59.368743 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:59.368758 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:56:59.368860 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:56:59.370215 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:56:59.370221 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:56:59.370747 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:56:59.371228 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:59.371243 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:59.372476 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:59.372478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:59.373383 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:56:59.373393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:56:59.374274 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:56:59.374278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:56:59.374477 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:56:59.375136 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:56:59.375151 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:56:59.377074 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:56:59.379440 (+ 2366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:59.379443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:59.379444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:59.379628 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:59.379936 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:56:59.379938 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:56:59.379940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:56:59.380011 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:56:59.380648 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:56:59.380651 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:56:59.380724 (+ 73us) client_negotiation.cc:769] Sending connection context
0809 07:56:59.380911 (+ 187us) client_negotiation.cc:240] Negotiation successful
0809 07:56:59.381230 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":298,"spinlock_wait_cycles":34304,"thread_start_us":73,"threads_started":1}
W20260809 07:56:59.383948 22019 leader_election.cc:341] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 79ece8ecd514451abbcebc8b74a37bf5 (127.28.112.3:44481): Not found: Tablet not found: 91263c725ebc47bc8a37a2b75e06e75e
W20260809 07:56:59.384123 22022 leader_election.cc:341] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9609c8bb0354680b799dd060579a25a (127.28.112.2:46401): Not found: Tablet not found: 91263c725ebc47bc8a37a2b75e06e75e
I20260809 07:56:59.384232 22022 leader_election.cc:302] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d1d92bb8bd84610b33f8546b515e0fd; no voters: 79ece8ecd514451abbcebc8b74a37bf5, f9609c8bb0354680b799dd060579a25a
I20260809 07:56:59.384510 22634 raft_consensus.cc:2726] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 07:56:59.949967 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:56:59.949563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:39623)
0809 07:56:59.949797 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0809 07:56:59.949850 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":73,"threads_started":1}
I20260809 07:57:00.070335 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:00.069991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:38913)
0809 07:57:00.070202 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:00.070254 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":67,"threads_started":1}
I20260809 07:57:00.273283 22647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:00.272895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:38887)
0809 07:57:00.273145 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:00.273199 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":82,"threads_started":1}
I20260809 07:57:00.951018 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:00.950611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:45539)
0809 07:57:00.950865 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:00.950924 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
I20260809 07:57:01.071321 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:01.070954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:55667)
0809 07:57:01.071180 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:01.071235 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
I20260809 07:57:01.171979 22650 raft_consensus.cc:482] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:57:01.172108 22650 raft_consensus.cc:504] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } } peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } }
I20260809 07:57:01.172503 22650 leader_election.cc:288] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79ece8ecd514451abbcebc8b74a37bf5 (127.28.112.3:44481), f9609c8bb0354680b799dd060579a25a (127.28.112.2:46401)
W20260809 07:57:01.173300 22019 leader_election.cc:341] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 79ece8ecd514451abbcebc8b74a37bf5 (127.28.112.3:44481): Not found: Tablet not found: 91263c725ebc47bc8a37a2b75e06e75e
W20260809 07:57:01.173418 22022 leader_election.cc:341] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9609c8bb0354680b799dd060579a25a (127.28.112.2:46401): Not found: Tablet not found: 91263c725ebc47bc8a37a2b75e06e75e
I20260809 07:57:01.173583 22022 leader_election.cc:302] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d1d92bb8bd84610b33f8546b515e0fd; no voters: 79ece8ecd514451abbcebc8b74a37bf5, f9609c8bb0354680b799dd060579a25a
I20260809 07:57:01.173758 22650 raft_consensus.cc:2726] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 07:57:01.274379 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:01.273979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:58723)
0809 07:57:01.274236 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:01.274291 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":93,"threads_started":1}
I20260809 07:57:01.952152 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:01.951722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:56033)
0809 07:57:01.951985 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:01.952040 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260809 07:57:02.072554 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:02.072020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:57533)
0809 07:57:02.072304 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:02.072454 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
I20260809 07:57:02.275465 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:02.275065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:44299)
0809 07:57:02.275317 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:02.275374 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
I20260809 07:57:02.953482 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:02.952986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:44527)
0809 07:57:02.953329 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:02.953386 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":98,"threads_started":1}
I20260809 07:57:03.073645 22657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:03.073254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:36443)
0809 07:57:03.073487 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:03.073541 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
I20260809 07:57:03.276536 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:03.276151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:45847)
0809 07:57:03.276398 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:03.276449 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":86,"threads_started":1}
I20260809 07:57:03.954650 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:03.954206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:53553)
0809 07:57:03.954486 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:03.954545 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":75,"threads_started":1}
I20260809 07:57:04.074818 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:04.074384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:58089)
0809 07:57:04.074626 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:04.074699 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260809 07:57:04.277560 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:04.277181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:50795)
0809 07:57:04.277416 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:04.277471 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260809 07:57:04.365743 22665 raft_consensus.cc:482] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:57:04.365873 22665 raft_consensus.cc:504] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } } peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } }
I20260809 07:57:04.366310 22665 leader_election.cc:288] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79ece8ecd514451abbcebc8b74a37bf5 (127.28.112.3:44481), f9609c8bb0354680b799dd060579a25a (127.28.112.2:46401)
W20260809 07:57:04.367108 22019 leader_election.cc:341] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 79ece8ecd514451abbcebc8b74a37bf5 (127.28.112.3:44481): Not found: Tablet not found: 91263c725ebc47bc8a37a2b75e06e75e
W20260809 07:57:04.367228 22022 leader_election.cc:341] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9609c8bb0354680b799dd060579a25a (127.28.112.2:46401): Not found: Tablet not found: 91263c725ebc47bc8a37a2b75e06e75e
I20260809 07:57:04.367295 22022 leader_election.cc:302] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d1d92bb8bd84610b33f8546b515e0fd; no voters: 79ece8ecd514451abbcebc8b74a37bf5, f9609c8bb0354680b799dd060579a25a
I20260809 07:57:04.367419 22665 raft_consensus.cc:2726] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 07:57:04.955726 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:04.955339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:41319)
0809 07:57:04.955572 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:04.955628 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":81,"threads_started":1}
I20260809 07:57:05.076002 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:05.075559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:56597)
0809 07:57:05.075857 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:05.075910 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":100,"threads_started":1}
I20260809 07:57:05.278649 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:05.278239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:53647)
0809 07:57:05.278508 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:05.278559 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":69,"threads_started":1}
I20260809 07:57:05.956755 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:05.956380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:54221)
0809 07:57:05.956610 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:05.956661 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":72,"threads_started":1}
I20260809 07:57:06.077100 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:06.076683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:52487)
0809 07:57:06.076944 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:06.077003 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260809 07:57:06.279675 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:06.279293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:59199)
0809 07:57:06.279520 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:06.279573 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
I20260809 07:57:06.957814 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:06.957407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:41619)
0809 07:57:06.957664 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:06.957716 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
I20260809 07:57:07.078181 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:07.077772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:50155)
0809 07:57:07.078033 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:07.078085 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":87,"threads_started":1}
I20260809 07:57:07.280666 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:07.280297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:59247)
0809 07:57:07.280527 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:07.280578 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":86,"threads_started":1}
I20260809 07:57:07.891350 22680 raft_consensus.cc:482] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:57:07.891474 22680 raft_consensus.cc:504] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } } peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } }
I20260809 07:57:07.891872 22680 leader_election.cc:288] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79ece8ecd514451abbcebc8b74a37bf5 (127.28.112.3:44481), f9609c8bb0354680b799dd060579a25a (127.28.112.2:46401)
W20260809 07:57:07.892623 22019 leader_election.cc:341] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 79ece8ecd514451abbcebc8b74a37bf5 (127.28.112.3:44481): Not found: Tablet not found: 91263c725ebc47bc8a37a2b75e06e75e
W20260809 07:57:07.892745 22022 leader_election.cc:341] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9609c8bb0354680b799dd060579a25a (127.28.112.2:46401): Not found: Tablet not found: 91263c725ebc47bc8a37a2b75e06e75e
I20260809 07:57:07.892827 22022 leader_election.cc:302] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d1d92bb8bd84610b33f8546b515e0fd; no voters: 79ece8ecd514451abbcebc8b74a37bf5, f9609c8bb0354680b799dd060579a25a
I20260809 07:57:07.892962 22680 raft_consensus.cc:2726] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 07:57:07.959029 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:07.958634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:41233)
0809 07:57:07.958883 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:07.958934 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260809 07:57:08.079224 22682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:08.078848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:43257)
0809 07:57:08.079079 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:08.079132 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":87,"threads_started":1}
I20260809 07:57:08.281754 22683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:08.281333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:53467)
0809 07:57:08.281602 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:08.281661 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":75,"threads_started":1}
I20260809 07:57:08.960129 22685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:08.959755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:51411)
0809 07:57:08.959983 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:08.960034 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":72,"threads_started":1}
I20260809 07:57:09.080266 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:09.079869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:33081)
0809 07:57:09.080107 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:09.080175 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":69,"threads_started":1}
I20260809 07:57:09.282800 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:09.282416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:32973)
0809 07:57:09.282659 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:09.282713 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260809 07:57:09.961437 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:09.960975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:46639)
0809 07:57:09.961267 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:09.961336 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":85,"threads_started":1}
I20260809 07:57:10.081586 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:10.081115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:43499)
0809 07:57:10.081414 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:10.081487 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":82,"threads_started":1}
I20260809 07:57:10.284022 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:10.283585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:43419)
0809 07:57:10.283863 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:10.283925 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
I20260809 07:57:10.962586 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:10.962164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:56343)
0809 07:57:10.962400 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:10.962458 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":76,"threads_started":1}
I20260809 07:57:11.082751 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:11.082334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:60281)
0809 07:57:11.082588 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:11.082651 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":90,"threads_started":1}
I20260809 07:57:11.285100 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:11.284705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:56125)
0809 07:57:11.284953 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:11.285008 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260809 07:57:11.963824 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:11.963399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:43277)
0809 07:57:11.963675 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:11.963730 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":78,"threads_started":1}
I20260809 07:57:12.083868 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:12.083457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:44823)
0809 07:57:12.083716 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:12.083772 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":82,"threads_started":1}
I20260809 07:57:12.286252 22702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:12.285795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:40859)
0809 07:57:12.286068 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:12.286145 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
I20260809 07:57:12.717027 22703 raft_consensus.cc:482] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:57:12.717166 22703 raft_consensus.cc:504] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ece8ecd514451abbcebc8b74a37bf5" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44481 } } peers { permanent_uuid: "f9609c8bb0354680b799dd060579a25a" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46401 } } peers { permanent_uuid: "5d1d92bb8bd84610b33f8546b515e0fd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 39299 } }
I20260809 07:57:12.717607 22703 leader_election.cc:288] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79ece8ecd514451abbcebc8b74a37bf5 (127.28.112.3:44481), f9609c8bb0354680b799dd060579a25a (127.28.112.2:46401)
W20260809 07:57:12.718361 22022 leader_election.cc:341] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9609c8bb0354680b799dd060579a25a (127.28.112.2:46401): Not found: Tablet not found: 91263c725ebc47bc8a37a2b75e06e75e
W20260809 07:57:12.718487 22019 leader_election.cc:341] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 79ece8ecd514451abbcebc8b74a37bf5 (127.28.112.3:44481): Not found: Tablet not found: 91263c725ebc47bc8a37a2b75e06e75e
I20260809 07:57:12.718600 22019 leader_election.cc:302] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d1d92bb8bd84610b33f8546b515e0fd; no voters: 79ece8ecd514451abbcebc8b74a37bf5, f9609c8bb0354680b799dd060579a25a
I20260809 07:57:12.718734 22703 raft_consensus.cc:2726] T 91263c725ebc47bc8a37a2b75e06e75e P 5d1d92bb8bd84610b33f8546b515e0fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 07:57:12.965034 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:12.964687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.1:58299)
0809 07:57:12.964895 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:12.964942 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
I20260809 07:57:13.085045 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:13.084637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.2:48853)
0809 07:57:13.084899 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:13.084952 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":91,"threads_started":1}
I20260809 07:57:13.270171 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 21934
I20260809 07:57:13.286104 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 22200
I20260809 07:57:13.293138 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:13.290642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:36865 (local address 127.28.112.3:34591)
0809 07:57:13.292964 (+ 2322us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:13.293028 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.62:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2231,"thread_start_us":1341,"threads_started":1}
I20260809 07:57:13.297739 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 22408
2026-08-09T07:57:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59312 ms)
[----------] 30 tests from SecurityITest (242174 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:13 dist-test-slave-vpvm krb5kdc[22710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:13 dist-test-slave-vpvm krb5kdc[22710](info): set up 2 sockets
Aug 09 07:57:13 dist-test-slave-vpvm krb5kdc[22710](info): commencing operation
krb5kdc: starting...
W20260809 07:57:14.447613 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:14 dist-test-slave-vpvm krb5kdc[22710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:57:14.560873 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:45447
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:45447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:14.670111 22730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:14.670385 22730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:14.670452 22730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:14.674496 22730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:14.674557 22730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:14.674584 22730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:14.674607 22730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:14.674638 22730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:14.679997 22730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:45447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:45447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:14.681070 22730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:14.682118 22730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:14.690374 22746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:14.690442 22730 server_base.cc:1034] running on GCE node
W20260809 07:57:14.690390 22742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:14.690374 22740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:14.691183 22730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:14.692215 22730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:14.693377 22730 hybrid_clock.cc:648] HybridClock initialized: now 1786262234693356 us; error 33 us; skew 500 ppm
Aug 09 07:57:14 dist-test-slave-vpvm krb5kdc[22710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262234, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:14.696116 22730 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:57:14.696977 22730 webserver.cc:460] Webserver started at http://127.28.112.62:40833/ using document root <none> and password file <none>
I20260809 07:57:14.697490 22730 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:14.697547 22730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:14.697713 22730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:14.699718 22730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "2cc6240f1f7443db91d1aa474ad1cb57"
format_stamp: "Formatted at 2026-08-09 07:57:14 on dist-test-slave-vpvm"
server_key: "2d847e905399460a430210258ab99c9e"
server_key_iv: "f59d8618118840f07f092bf6f6382e6c"
server_key_version: "encryptionkey@0"
I20260809 07:57:14.700239 22730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "2cc6240f1f7443db91d1aa474ad1cb57"
format_stamp: "Formatted at 2026-08-09 07:57:14 on dist-test-slave-vpvm"
server_key: "2d847e905399460a430210258ab99c9e"
server_key_iv: "f59d8618118840f07f092bf6f6382e6c"
server_key_version: "encryptionkey@0"
I20260809 07:57:14.703971 22730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 07:57:14.706301 22754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:14.707475 22730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:57:14.707580 22730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "2cc6240f1f7443db91d1aa474ad1cb57"
format_stamp: "Formatted at 2026-08-09 07:57:14 on dist-test-slave-vpvm"
server_key: "2d847e905399460a430210258ab99c9e"
server_key_iv: "f59d8618118840f07f092bf6f6382e6c"
server_key_version: "encryptionkey@0"
I20260809 07:57:14.707670 22730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:14.719890 22730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:14.722965 22730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:14.723119 22730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:14.730046 22730 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:45447
I20260809 07:57:14.730072 22831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:45447 every 8 connection(s)
I20260809 07:57:14.731009 22730 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:14.733811 22832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:14.736459 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 22730
I20260809 07:57:14.736569 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:14.736841 29120 external_mini_cluster.cc:1428] Setting key 07ae54ba79b36c2069283a0fa093b6b4
I20260809 07:57:14.739636 22832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57: Bootstrap starting.
I20260809 07:57:14.741741 22832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:14.742635 22832 log.cc:824] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:14.744931 22832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57: No bootstrap required, opened a new log
Aug 09 07:57:14 dist-test-slave-vpvm krb5kdc[22710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:57:14.748039 22832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cc6240f1f7443db91d1aa474ad1cb57" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45447 } }
I20260809 07:57:14.748262 22832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:14.748319 22832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc6240f1f7443db91d1aa474ad1cb57, State: Initialized, Role: FOLLOWER
I20260809 07:57:14.748761 22832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cc6240f1f7443db91d1aa474ad1cb57" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45447 } }
I20260809 07:57:14.748893 22832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:14.748957 22832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:14.749056 22832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:14.749905 22832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cc6240f1f7443db91d1aa474ad1cb57" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45447 } }
I20260809 07:57:14.750290 22832 leader_election.cc:302] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cc6240f1f7443db91d1aa474ad1cb57; no voters:
I20260809 07:57:14.750684 22832 leader_election.cc:288] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:14.750910 22840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:14.751168 22840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57 [term 1 LEADER]: Becoming Leader. State: Replica: 2cc6240f1f7443db91d1aa474ad1cb57, State: Running, Role: LEADER
I20260809 07:57:14.751544 22840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cc6240f1f7443db91d1aa474ad1cb57" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45447 } }
I20260809 07:57:14.751824 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:14.738277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.28.112.62:45447)
0809 07:57:14.738727 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:14.738741 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:14.738775 (+ 34us) server_negotiation.cc:407] Connection header received
0809 07:57:14.739359 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:14.739376 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:14.739697 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:14.740013 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:14.740827 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:14.741814 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:14.743018 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:14.743653 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:14.743778 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:14.746494 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:14.746499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:14.746514 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:14.746541 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:14.748675 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:14.749155 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:14.749161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:14.749168 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:14.749286 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:14.749560 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:14.749564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:14.749565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:14.750003 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:14.750192 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:14.750476 (+ 284us) server_negotiation.cc:299] Negotiation successful
0809 07:57:14.750778 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":146,"threads_started":1}
I20260809 07:57:14.751909 22832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:14.753185 22843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cc6240f1f7443db91d1aa474ad1cb57. Latest consensus state: current_term: 1 leader_uuid: "2cc6240f1f7443db91d1aa474ad1cb57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cc6240f1f7443db91d1aa474ad1cb57" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45447 } } }
I20260809 07:57:14.753305 22843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:14.753608 22841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cc6240f1f7443db91d1aa474ad1cb57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cc6240f1f7443db91d1aa474ad1cb57" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45447 } } }
I20260809 07:57:14.753697 22841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57 [sys.catalog]: This master's current role is: LEADER
W20260809 07:57:14.756685 22860 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:57:14.756752 22860 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:57:14.756825 22853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:14.757525 22853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:14.762981 22853 catalog_manager.cc:1347] Generated new cluster ID: 8f3e0e444f5e4df3995732770b29cd4d
I20260809 07:57:14.763044 22853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:14.777787 22853 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:14.779067 22853 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:14.786459 22853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57: Generated new TSK 0
I20260809 07:57:14.787113 22853 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
I20260809 07:57:14.821091 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:45447
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:14.933259 22869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:14.933511 22869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:14.933553 22869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:14.937501 22869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:14.937556 22869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:14.937611 22869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:57:14.942777 22869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:45447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:14.943787 22869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:14.944756 22869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:14.953933 22875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:14.953933 22877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:14.954033 22869 server_base.cc:1034] running on GCE node
W20260809 07:57:14.954181 22881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:14.954559 22869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:14.955194 22869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:14.956348 22869 hybrid_clock.cc:648] HybridClock initialized: now 1786262234956334 us; error 30 us; skew 500 ppm
Aug 09 07:57:14 dist-test-slave-vpvm krb5kdc[22710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262234, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:14.958959 22869 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:57:14.959766 22869 webserver.cc:460] Webserver started at http://127.28.112.1:37047/ using document root <none> and password file <none>
I20260809 07:57:14.960294 22869 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:14.960358 22869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:14.960556 22869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:14.962565 22869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "34f56e1177fe4d4a820253f27b0defde"
format_stamp: "Formatted at 2026-08-09 07:57:14 on dist-test-slave-vpvm"
server_key: "d25830d5cf006965165848d984049731"
server_key_iv: "7f7ce8e59d61ba6f21fd632743adda79"
server_key_version: "encryptionkey@0"
I20260809 07:57:14.963080 22869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "34f56e1177fe4d4a820253f27b0defde"
format_stamp: "Formatted at 2026-08-09 07:57:14 on dist-test-slave-vpvm"
server_key: "d25830d5cf006965165848d984049731"
server_key_iv: "7f7ce8e59d61ba6f21fd632743adda79"
server_key_version: "encryptionkey@0"
I20260809 07:57:14.966632 22869 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 07:57:14.968942 22893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:14.970099 22869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260809 07:57:14.970212 22869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "34f56e1177fe4d4a820253f27b0defde"
format_stamp: "Formatted at 2026-08-09 07:57:14 on dist-test-slave-vpvm"
server_key: "d25830d5cf006965165848d984049731"
server_key_iv: "7f7ce8e59d61ba6f21fd632743adda79"
server_key_version: "encryptionkey@0"
I20260809 07:57:14.970302 22869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:14.997817 22869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:15.000833 22869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:15.001025 22869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:15.001683 22869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:57:15.002604 22869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:57:15.002660 22869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:15.002717 22869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:57:15.002767 22869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:15.011335 22869 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:35577
I20260809 07:57:15.011369 23070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:35577 every 8 connection(s)
I20260809 07:57:15.012254 22869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:57:15.018667 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 22869
I20260809 07:57:15.018814 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:57:15.019132 29120 external_mini_cluster.cc:1428] Setting key f8721affe52a434f3c7262f3ae2ebd1b
Aug 09 07:57:15 dist-test-slave-vpvm krb5kdc[22710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262234, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:57:15.025625 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.014052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:54041 (local address 127.28.112.62:45447)
0809 07:57:15.014212 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:15.014217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:15.015308 (+ 1091us) server_negotiation.cc:407] Connection header received
0809 07:57:15.016126 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:15.016129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:15.016180 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:15.016242 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:15.017341 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.017880 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.019353 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.019598 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.019707 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:15.022204 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:15.022209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:15.022212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:15.022241 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:15.023620 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:15.024142 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:15.024144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:15.024145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:15.024216 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:15.024545 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:15.024547 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:15.024548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:15.024705 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:15.024772 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:15.025382 (+ 610us) server_negotiation.cc:299] Negotiation successful
0809 07:57:15.025501 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 07:57:15.026726 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.014497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:45447 (local address 127.28.112.1:54041)
0809 07:57:15.015162 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:15.015203 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:15.015945 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:15.016391 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:15.016398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:15.016738 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:15.017225 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:15.017239 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.018025 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.018028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:15.019238 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:15.019249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.019772 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.019776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:15.019938 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:15.020507 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:15.020524 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:15.022017 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:15.023714 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:15.023720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:15.023734 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:15.024052 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:15.024292 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:15.024296 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:15.024297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:15.024467 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:15.024794 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:15.024800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:15.025123 (+ 323us) client_negotiation.cc:769] Sending connection context
0809 07:57:15.025330 (+ 207us) client_negotiation.cc:240] Negotiation successful
0809 07:57:15.025675 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":121,"threads_started":1}
I20260809 07:57:15.028190 23071 heartbeater.cc:347] Connected to a master server at 127.28.112.62:45447
I20260809 07:57:15.028451 23071 heartbeater.cc:464] Registering TS with master...
I20260809 07:57:15.028957 23071 heartbeater.cc:511] Master 127.28.112.62:45447 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.112.2@KRBTEST.COM; defaulting to no policy
I20260809 07:57:15.030550 22778 ts_manager.cc:194] Registered new tserver with Master: 34f56e1177fe4d4a820253f27b0defde (127.28.112.1:35577)
I20260809 07:57:15.031651 22778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:54041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
I20260809 07:57:15.046355 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:45447
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:15.158560 23083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:15.158799 23083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:15.158849 23083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:15.162885 23083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:15.162947 23083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:15.163028 23083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:57:15.168177 23083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:45447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:15.169152 23083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:15.170192 23083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:15.179497 23090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:15.179471 23089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:15.179468 23092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:15.179672 23083 server_base.cc:1034] running on GCE node
I20260809 07:57:15.180161 23083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:15.180737 23083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:15.181890 23083 hybrid_clock.cc:648] HybridClock initialized: now 1786262235181872 us; error 34 us; skew 500 ppm
Aug 09 07:57:15 dist-test-slave-vpvm krb5kdc[22710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262235, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:15.184496 23083 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:57:15.185322 23083 webserver.cc:460] Webserver started at http://127.28.112.2:33913/ using document root <none> and password file <none>
I20260809 07:57:15.185856 23083 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:15.185922 23083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:15.186152 23083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:15.188123 23083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "44c6ea408cea4954949ba0d343e72f8f"
format_stamp: "Formatted at 2026-08-09 07:57:15 on dist-test-slave-vpvm"
server_key: "b3be32b42d08f822610ad867e2d48c23"
server_key_iv: "8d615ed46db6c04758d02fd0fe14622f"
server_key_version: "encryptionkey@0"
I20260809 07:57:15.188650 23083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "44c6ea408cea4954949ba0d343e72f8f"
format_stamp: "Formatted at 2026-08-09 07:57:15 on dist-test-slave-vpvm"
server_key: "b3be32b42d08f822610ad867e2d48c23"
server_key_iv: "8d615ed46db6c04758d02fd0fe14622f"
server_key_version: "encryptionkey@0"
I20260809 07:57:15.192242 23083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 07:57:15.194576 23101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:15.195742 23083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:57:15.195853 23083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "44c6ea408cea4954949ba0d343e72f8f"
format_stamp: "Formatted at 2026-08-09 07:57:15 on dist-test-slave-vpvm"
server_key: "b3be32b42d08f822610ad867e2d48c23"
server_key_iv: "8d615ed46db6c04758d02fd0fe14622f"
server_key_version: "encryptionkey@0"
I20260809 07:57:15.195946 23083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:15.220297 23083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:15.223042 23083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:15.223237 23083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:15.223812 23083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:57:15.224632 23083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:57:15.224680 23083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:15.224733 23083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:57:15.224757 23083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:15.233120 23083 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:44243
I20260809 07:57:15.233139 23279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:44243 every 8 connection(s)
I20260809 07:57:15.233976 23083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
Aug 09 07:57:15 dist-test-slave-vpvm krb5kdc[22710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262235, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:57:15.242558 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 23083
I20260809 07:57:15.242677 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:57:15.242938 29120 external_mini_cluster.cc:1428] Setting key 9994189e0722d2084b20f24dc8fea609
I20260809 07:57:15.246637 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.235695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:56167 (local address 127.28.112.62:45447)
0809 07:57:15.237015 (+ 1320us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:15.237019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:15.237035 (+ 16us) server_negotiation.cc:407] Connection header received
0809 07:57:15.237793 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:15.237796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:15.237842 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:15.237910 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:15.239028 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.239545 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.240585 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.240805 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.240873 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:15.243193 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:15.243197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:15.243200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:15.243228 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:15.244728 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:15.245178 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:15.245182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:15.245184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:15.245276 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:15.245548 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:15.245550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:15.245552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:15.245764 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:15.245851 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:15.246355 (+ 504us) server_negotiation.cc:299] Negotiation successful
0809 07:57:15.246493 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1229}
I20260809 07:57:15.247604 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.236209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:45447 (local address 127.28.112.2:56167)
0809 07:57:15.236852 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:15.236894 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:15.237611 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:15.238093 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:15.238100 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:15.238429 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:15.238903 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:15.238916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.239648 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.239650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:15.240497 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:15.240505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.240913 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.240916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:15.241056 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:15.241531 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:15.241547 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:15.243005 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:15.244833 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:15.244838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:15.244849 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:15.245087 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:15.245353 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:15.245355 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:15.245356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:15.245475 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:15.245851 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:15.245855 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:15.246128 (+ 273us) client_negotiation.cc:769] Sending connection context
0809 07:57:15.246286 (+ 158us) client_negotiation.cc:240] Negotiation successful
0809 07:57:15.246613 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":104,"threads_started":1}
I20260809 07:57:15.248797 23280 heartbeater.cc:347] Connected to a master server at 127.28.112.62:45447
I20260809 07:57:15.249025 23280 heartbeater.cc:464] Registering TS with master...
I20260809 07:57:15.249477 23280 heartbeater.cc:511] Master 127.28.112.62:45447 requested a full tablet report, sending...
I20260809 07:57:15.250506 22778 ts_manager.cc:194] Registered new tserver with Master: 44c6ea408cea4954949ba0d343e72f8f (127.28.112.2:44243)
I20260809 07:57:15.251187 22778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:56167
WARNING: no policy specified for kudu/127.28.112.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
I20260809 07:57:15.270206 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:45447
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:15.379354 23286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:15.379575 23286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:15.379613 23286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:15.383538 23286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:15.383584 23286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:15.383642 23286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:57:15.388713 23286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:45447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:15.389597 23286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:15.390529 23286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:15.399326 23294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:15.399370 23299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:15.399523 23286 server_base.cc:1034] running on GCE node
W20260809 07:57:15.399331 23293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:15.400015 23286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:15.400599 23286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:15.401765 23286 hybrid_clock.cc:648] HybridClock initialized: now 1786262235401736 us; error 38 us; skew 500 ppm
Aug 09 07:57:15 dist-test-slave-vpvm krb5kdc[22710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262235, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:15.404456 23286 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:57:15.405244 23286 webserver.cc:460] Webserver started at http://127.28.112.3:40587/ using document root <none> and password file <none>
I20260809 07:57:15.405755 23286 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:15.405812 23286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:15.406002 23286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:15.407925 23286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "8325eed77daf48f7b8e31a875f3d809e"
format_stamp: "Formatted at 2026-08-09 07:57:15 on dist-test-slave-vpvm"
server_key: "2882289c776e5c51bc1b6c4ff7d165e1"
server_key_iv: "2542eb5fbaff40cb74e7e4d71979a0cb"
server_key_version: "encryptionkey@0"
I20260809 07:57:15.408409 23286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "8325eed77daf48f7b8e31a875f3d809e"
format_stamp: "Formatted at 2026-08-09 07:57:15 on dist-test-slave-vpvm"
server_key: "2882289c776e5c51bc1b6c4ff7d165e1"
server_key_iv: "2542eb5fbaff40cb74e7e4d71979a0cb"
server_key_version: "encryptionkey@0"
I20260809 07:57:15.411978 23286 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 07:57:15.414249 23313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:15.415351 23286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 07:57:15.415468 23286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "8325eed77daf48f7b8e31a875f3d809e"
format_stamp: "Formatted at 2026-08-09 07:57:15 on dist-test-slave-vpvm"
server_key: "2882289c776e5c51bc1b6c4ff7d165e1"
server_key_iv: "2542eb5fbaff40cb74e7e4d71979a0cb"
server_key_version: "encryptionkey@0"
I20260809 07:57:15.415561 23286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:15.446239 23286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:15.448786 23286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:15.448947 23286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:15.449489 23286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:57:15.450313 23286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:57:15.450361 23286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:15.450415 23286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:57:15.450444 23286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:15.458890 23286 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:42169
I20260809 07:57:15.458940 23476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:42169 every 8 connection(s)
I20260809 07:57:15.459719 23286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:57:15.466123 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 23286
I20260809 07:57:15.466261 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:57:15.466595 29120 external_mini_cluster.cc:1428] Setting key 02a802b65d44767b96314665ddfb4fcb
Aug 09 07:57:15 dist-test-slave-vpvm krb5kdc[22710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262235, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:57:15.471702 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.461374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:54305 (local address 127.28.112.62:45447)
0809 07:57:15.461513 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:15.461517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:15.462491 (+ 974us) server_negotiation.cc:407] Connection header received
0809 07:57:15.463294 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:15.463296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:15.463342 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:15.463410 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:15.464456 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.464964 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.466042 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.466368 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.466478 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:15.468656 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:15.468661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:15.468664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:15.468687 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:15.469751 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:15.470264 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:15.470267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:15.470269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:15.470350 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:15.470668 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:15.470672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:15.470673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:15.470850 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:15.470935 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:15.471410 (+ 475us) server_negotiation.cc:299] Negotiation successful
0809 07:57:15.471534 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 07:57:15.472385 23479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.461711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:45447 (local address 127.28.112.3:54305)
0809 07:57:15.462358 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:15.462398 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:15.463109 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:15.463555 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:15.463561 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:15.463888 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:15.464345 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:15.464358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.465048 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.465050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:15.465890 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:15.465899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.466506 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.466509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:15.466644 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:15.467154 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:15.467170 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:15.468494 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:15.469857 (+ 1363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:15.469863 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:15.469877 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:15.470171 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:15.470432 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:15.470435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:15.470437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:15.470565 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:15.470946 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:15.470953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:15.471181 (+ 228us) client_negotiation.cc:769] Sending connection context
0809 07:57:15.471331 (+ 150us) client_negotiation.cc:240] Negotiation successful
0809 07:57:15.471622 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":103,"threads_started":1}
I20260809 07:57:15.473402 23477 heartbeater.cc:347] Connected to a master server at 127.28.112.62:45447
I20260809 07:57:15.473646 23477 heartbeater.cc:464] Registering TS with master...
I20260809 07:57:15.474146 23477 heartbeater.cc:511] Master 127.28.112.62:45447 requested a full tablet report, sending...
I20260809 07:57:15.475183 22778 ts_manager.cc:194] Registered new tserver with Master: 8325eed77daf48f7b8e31a875f3d809e (127.28.112.3:42169)
I20260809 07:57:15.475723 22778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:54305
I20260809 07:57:15.479919 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 07:57:15.488391 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.481835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.28.112.62:45447)
0809 07:57:15.481930 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:15.481933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:15.482171 (+ 238us) server_negotiation.cc:407] Connection header received
0809 07:57:15.482267 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:15.482269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:15.482306 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:15.482366 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:15.483020 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.483496 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.484311 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.484502 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.484564 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:15.485770 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:15.485773 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:15.485775 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:15.485797 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:15.487016 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:15.487377 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:15.487380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:15.487381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:15.487445 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:15.487691 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:15.487694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:15.487696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:15.487931 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:15.488032 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:15.488200 (+ 168us) server_negotiation.cc:299] Negotiation successful
0809 07:57:15.488286 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 07:57:15.490845 22778 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:57:15.493687 22778 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:57:15.507236 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.502502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39738 (local address 127.28.112.2:44243)
0809 07:57:15.502862 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:15.502871 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:15.502889 (+ 18us) server_negotiation.cc:407] Connection header received
0809 07:57:15.502948 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:15.502951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:15.503076 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:15.503209 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:15.503471 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.504452 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.505728 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.506583 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.506704 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:15.506846 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:15.506999 (+ 153us) server_negotiation.cc:299] Negotiation successful
0809 07:57:15.507090 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":95,"threads_started":1}
I20260809 07:57:15.507231 23493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.501402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:44243 (local address 127.0.0.1:39738)
0809 07:57:15.502290 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:15.502303 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:15.502377 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:15.503221 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:15.503225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:15.503244 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:15.503337 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:15.503343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.504570 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.504573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:15.505606 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:15.505616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.506717 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.506721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:15.506823 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:15.506867 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 07:57:15.506966 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 07:57:15.507078 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260809 07:57:15.507408 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.501282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:35577 (local address 127.0.0.1:56954)
0809 07:57:15.501763 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:15.501813 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:15.501943 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:15.502619 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:15.502661 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:15.502689 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:15.502828 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:15.502834 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.504303 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.504306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:15.505675 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:15.505687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.506926 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.506930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:15.507040 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:15.507065 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 07:57:15.507147 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 07:57:15.507214 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":51,"thread_start_us":81,"threads_started":1}
I20260809 07:57:15.507429 23497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.501438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56954 (local address 127.28.112.1:35577)
0809 07:57:15.501858 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:15.501873 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:15.501897 (+ 24us) server_negotiation.cc:407] Connection header received
0809 07:57:15.502174 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:15.502181 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:15.502394 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:15.502909 (+ 515us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:15.503045 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.504172 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.505806 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.506583 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.506704 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:15.506846 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:15.507203 (+ 357us) server_negotiation.cc:299] Negotiation successful
0809 07:57:15.507298 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":127,"threads_started":1}
I20260809 07:57:15.508064 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.502016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:42169 (local address 127.0.0.1:59126)
0809 07:57:15.502717 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:15.502730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:15.502811 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:15.503806 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:15.503810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:15.503826 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:15.503922 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:15.503928 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.505200 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.505203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:15.506128 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:15.506137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.506926 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.506930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:15.507785 (+ 855us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:15.507805 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 07:57:15.507878 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 07:57:15.507945 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":775,"thread_start_us":93,"threads_started":1}
I20260809 07:57:15.508180 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.502177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59126 (local address 127.28.112.3:42169)
0809 07:57:15.503453 (+ 1276us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:15.503464 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:15.503489 (+ 25us) server_negotiation.cc:407] Connection header received
0809 07:57:15.503549 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:15.503554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:15.503687 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:15.503811 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:15.504016 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.505045 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.506233 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.506807 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.506905 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:15.507033 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:15.507947 (+ 914us) server_negotiation.cc:299] Negotiation successful
0809 07:57:15.508039 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1179,"thread_start_us":91,"threads_started":1}
I20260809 07:57:15.510402 22965 tablet_service.cc:1467] Processing CreateTablet for tablet 4edf402bf8914f30888ba2d4503d6165 (DEFAULT_TABLE table=test-table [id=4c206129084447b3b0e2d5dab89bf3aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:57:15.510721 23182 tablet_service.cc:1467] Processing CreateTablet for tablet 4edf402bf8914f30888ba2d4503d6165 (DEFAULT_TABLE table=test-table [id=4c206129084447b3b0e2d5dab89bf3aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:57:15.511620 22965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4edf402bf8914f30888ba2d4503d6165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:15.511647 23182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4edf402bf8914f30888ba2d4503d6165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:15.511672 23382 tablet_service.cc:1467] Processing CreateTablet for tablet 4edf402bf8914f30888ba2d4503d6165 (DEFAULT_TABLE table=test-table [id=4c206129084447b3b0e2d5dab89bf3aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:57:15.512583 23382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4edf402bf8914f30888ba2d4503d6165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:15.517776 23503 tablet_bootstrap.cc:492] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f: Bootstrap starting.
I20260809 07:57:15.519294 23504 tablet_bootstrap.cc:492] T 4edf402bf8914f30888ba2d4503d6165 P 34f56e1177fe4d4a820253f27b0defde: Bootstrap starting.
I20260809 07:57:15.519729 23505 tablet_bootstrap.cc:492] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e: Bootstrap starting.
I20260809 07:57:15.520745 23503 tablet_bootstrap.cc:654] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:15.521376 23504 tablet_bootstrap.cc:654] T 4edf402bf8914f30888ba2d4503d6165 P 34f56e1177fe4d4a820253f27b0defde: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:15.521471 23503 log.cc:824] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:15.521692 23505 tablet_bootstrap.cc:654] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:15.522651 23504 log.cc:824] T 4edf402bf8914f30888ba2d4503d6165 P 34f56e1177fe4d4a820253f27b0defde: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:15.522820 23505 log.cc:824] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:15.523372 23503 tablet_bootstrap.cc:492] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f: No bootstrap required, opened a new log
I20260809 07:57:15.523558 23503 ts_tablet_manager.cc:1397] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 07:57:15.524586 23505 tablet_bootstrap.cc:492] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e: No bootstrap required, opened a new log
I20260809 07:57:15.524583 23504 tablet_bootstrap.cc:492] T 4edf402bf8914f30888ba2d4503d6165 P 34f56e1177fe4d4a820253f27b0defde: No bootstrap required, opened a new log
I20260809 07:57:15.524760 23505 ts_tablet_manager.cc:1397] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 07:57:15.524760 23504 ts_tablet_manager.cc:1397] T 4edf402bf8914f30888ba2d4503d6165 P 34f56e1177fe4d4a820253f27b0defde: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 07:57:15.526594 23503 raft_consensus.cc:348] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f56e1177fe4d4a820253f27b0defde" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35577 } } peers { permanent_uuid: "44c6ea408cea4954949ba0d343e72f8f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 44243 } } peers { permanent_uuid: "8325eed77daf48f7b8e31a875f3d809e" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 42169 } }
I20260809 07:57:15.526814 23503 raft_consensus.cc:374] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:15.526867 23503 raft_consensus.cc:729] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c6ea408cea4954949ba0d343e72f8f, State: Initialized, Role: FOLLOWER
I20260809 07:57:15.527298 23503 consensus_queue.cc:260] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f56e1177fe4d4a820253f27b0defde" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35577 } } peers { permanent_uuid: "44c6ea408cea4954949ba0d343e72f8f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 44243 } } peers { permanent_uuid: "8325eed77daf48f7b8e31a875f3d809e" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 42169 } }
I20260809 07:57:15.527920 23280 heartbeater.cc:503] Master 127.28.112.62:45447 was elected leader, sending a full tablet report...
I20260809 07:57:15.528260 23503 ts_tablet_manager.cc:1428] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 07:57:15.528358 23504 raft_consensus.cc:348] T 4edf402bf8914f30888ba2d4503d6165 P 34f56e1177fe4d4a820253f27b0defde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f56e1177fe4d4a820253f27b0defde" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35577 } } peers { permanent_uuid: "44c6ea408cea4954949ba0d343e72f8f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 44243 } } peers { permanent_uuid: "8325eed77daf48f7b8e31a875f3d809e" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 42169 } }
I20260809 07:57:15.528358 23505 raft_consensus.cc:348] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f56e1177fe4d4a820253f27b0defde" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35577 } } peers { permanent_uuid: "44c6ea408cea4954949ba0d343e72f8f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 44243 } } peers { permanent_uuid: "8325eed77daf48f7b8e31a875f3d809e" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 42169 } }
I20260809 07:57:15.528590 23505 raft_consensus.cc:374] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:15.528599 23504 raft_consensus.cc:374] T 4edf402bf8914f30888ba2d4503d6165 P 34f56e1177fe4d4a820253f27b0defde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:15.528640 23504 raft_consensus.cc:729] T 4edf402bf8914f30888ba2d4503d6165 P 34f56e1177fe4d4a820253f27b0defde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f56e1177fe4d4a820253f27b0defde, State: Initialized, Role: FOLLOWER
I20260809 07:57:15.528647 23505 raft_consensus.cc:729] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8325eed77daf48f7b8e31a875f3d809e, State: Initialized, Role: FOLLOWER
I20260809 07:57:15.529071 23504 consensus_queue.cc:260] T 4edf402bf8914f30888ba2d4503d6165 P 34f56e1177fe4d4a820253f27b0defde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f56e1177fe4d4a820253f27b0defde" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35577 } } peers { permanent_uuid: "44c6ea408cea4954949ba0d343e72f8f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 44243 } } peers { permanent_uuid: "8325eed77daf48f7b8e31a875f3d809e" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 42169 } }
I20260809 07:57:15.529071 23505 consensus_queue.cc:260] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f56e1177fe4d4a820253f27b0defde" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35577 } } peers { permanent_uuid: "44c6ea408cea4954949ba0d343e72f8f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 44243 } } peers { permanent_uuid: "8325eed77daf48f7b8e31a875f3d809e" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 42169 } }
I20260809 07:57:15.529655 23071 heartbeater.cc:503] Master 127.28.112.62:45447 was elected leader, sending a full tablet report...
I20260809 07:57:15.529928 23477 heartbeater.cc:503] Master 127.28.112.62:45447 was elected leader, sending a full tablet report...
I20260809 07:57:15.529999 23504 ts_tablet_manager.cc:1428] T 4edf402bf8914f30888ba2d4503d6165 P 34f56e1177fe4d4a820253f27b0defde: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260809 07:57:15.531519 23505 ts_tablet_manager.cc:1428] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260809 07:57:15.642782 23509 raft_consensus.cc:482] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:57:15.642992 23509 raft_consensus.cc:504] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f56e1177fe4d4a820253f27b0defde" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35577 } } peers { permanent_uuid: "44c6ea408cea4954949ba0d343e72f8f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 44243 } } peers { permanent_uuid: "8325eed77daf48f7b8e31a875f3d809e" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 42169 } }
I20260809 07:57:15.644191 23509 leader_election.cc:288] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34f56e1177fe4d4a820253f27b0defde (127.28.112.1:35577), 8325eed77daf48f7b8e31a875f3d809e (127.28.112.3:42169)
I20260809 07:57:15.648908 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.644431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:35577 (local address 127.28.112.2:35963)
0809 07:57:15.644562 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:15.644576 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:15.644681 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:15.645009 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:15.645012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:15.645033 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:15.645167 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:15.645173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.646193 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.646196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:15.647490 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:15.647503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.648346 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.648349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:15.648543 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:15.648579 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 07:57:15.648658 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 07:57:15.648750 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":88}
I20260809 07:57:15.648924 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.644850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:59551 (local address 127.28.112.3:42169)
0809 07:57:15.644971 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:15.644975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:15.645131 (+ 156us) server_negotiation.cc:407] Connection header received
0809 07:57:15.645224 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:15.645228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:15.645273 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:15.645351 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:15.645546 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.646270 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.647642 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.648234 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.648347 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:15.648399 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:15.648721 (+ 322us) server_negotiation.cc:299] Negotiation successful
0809 07:57:15.648798 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 07:57:15.648927 23497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.644557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:35963 (local address 127.28.112.1:35577)
0809 07:57:15.644680 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:15.644686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:15.644705 (+ 19us) server_negotiation.cc:407] Connection header received
0809 07:57:15.644769 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:15.644772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:15.644827 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:15.644916 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:15.645267 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.646099 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.647641 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.648245 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.648347 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:15.648400 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:15.648721 (+ 321us) server_negotiation.cc:299] Negotiation successful
0809 07:57:15.648799 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 07:57:15.648916 23514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.644699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:42169 (local address 127.28.112.2:59551)
0809 07:57:15.645046 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:15.645058 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:15.645148 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:15.645354 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:15.645356 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:15.645369 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:15.645456 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:15.645460 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.646356 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.646359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:15.647525 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:15.647533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.648345 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.648349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:15.648548 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:15.648579 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 07:57:15.648657 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 07:57:15.648751 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":91,"thread_start_us":136,"threads_started":1}
I20260809 07:57:15.649636 22995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4edf402bf8914f30888ba2d4503d6165" candidate_uuid: "44c6ea408cea4954949ba0d343e72f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34f56e1177fe4d4a820253f27b0defde" is_pre_election: true
I20260809 07:57:15.649768 23406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4edf402bf8914f30888ba2d4503d6165" candidate_uuid: "44c6ea408cea4954949ba0d343e72f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8325eed77daf48f7b8e31a875f3d809e" is_pre_election: true
I20260809 07:57:15.649992 22995 raft_consensus.cc:2454] T 4edf402bf8914f30888ba2d4503d6165 P 34f56e1177fe4d4a820253f27b0defde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c6ea408cea4954949ba0d343e72f8f in term 0.
I20260809 07:57:15.650041 23406 raft_consensus.cc:2454] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c6ea408cea4954949ba0d343e72f8f in term 0.
I20260809 07:57:15.650471 23102 leader_election.cc:302] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34f56e1177fe4d4a820253f27b0defde, 44c6ea408cea4954949ba0d343e72f8f; no voters:
I20260809 07:57:15.650727 23509 raft_consensus.cc:2781] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:57:15.650813 23509 raft_consensus.cc:482] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:57:15.650862 23509 raft_consensus.cc:3037] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:15.652021 23509 raft_consensus.cc:504] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f56e1177fe4d4a820253f27b0defde" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35577 } } peers { permanent_uuid: "44c6ea408cea4954949ba0d343e72f8f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 44243 } } peers { permanent_uuid: "8325eed77daf48f7b8e31a875f3d809e" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 42169 } }
I20260809 07:57:15.652320 23509 leader_election.cc:288] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [CANDIDATE]: Term 1 election: Requested vote from peers 34f56e1177fe4d4a820253f27b0defde (127.28.112.1:35577), 8325eed77daf48f7b8e31a875f3d809e (127.28.112.3:42169)
I20260809 07:57:15.652688 23406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4edf402bf8914f30888ba2d4503d6165" candidate_uuid: "44c6ea408cea4954949ba0d343e72f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8325eed77daf48f7b8e31a875f3d809e"
I20260809 07:57:15.652714 22995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4edf402bf8914f30888ba2d4503d6165" candidate_uuid: "44c6ea408cea4954949ba0d343e72f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34f56e1177fe4d4a820253f27b0defde"
I20260809 07:57:15.652793 23406 raft_consensus.cc:3037] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:15.652817 22995 raft_consensus.cc:3037] T 4edf402bf8914f30888ba2d4503d6165 P 34f56e1177fe4d4a820253f27b0defde [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:15.653864 23406 raft_consensus.cc:2454] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c6ea408cea4954949ba0d343e72f8f in term 1.
I20260809 07:57:15.653934 22995 raft_consensus.cc:2454] T 4edf402bf8914f30888ba2d4503d6165 P 34f56e1177fe4d4a820253f27b0defde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c6ea408cea4954949ba0d343e72f8f in term 1.
I20260809 07:57:15.654165 23102 leader_election.cc:302] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44c6ea408cea4954949ba0d343e72f8f, 8325eed77daf48f7b8e31a875f3d809e; no voters:
I20260809 07:57:15.654336 23509 raft_consensus.cc:2781] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:15.654556 23509 raft_consensus.cc:686] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [term 1 LEADER]: Becoming Leader. State: Replica: 44c6ea408cea4954949ba0d343e72f8f, State: Running, Role: LEADER
I20260809 07:57:15.654855 23509 consensus_queue.cc:237] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f56e1177fe4d4a820253f27b0defde" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35577 } } peers { permanent_uuid: "44c6ea408cea4954949ba0d343e72f8f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 44243 } } peers { permanent_uuid: "8325eed77daf48f7b8e31a875f3d809e" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 42169 } }
I20260809 07:57:15.658042 22775 catalog_manager.cc:5495] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 44c6ea408cea4954949ba0d343e72f8f (127.28.112.2). New cstate: current_term: 1 leader_uuid: "44c6ea408cea4954949ba0d343e72f8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f56e1177fe4d4a820253f27b0defde" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c6ea408cea4954949ba0d343e72f8f" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 44243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8325eed77daf48f7b8e31a875f3d809e" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 42169 } health_report { overall_health: UNKNOWN } } }
I20260809 07:57:15.695678 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.692532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39750 (local address 127.28.112.2:44243)
0809 07:57:15.692665 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:15.692669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:15.692682 (+ 13us) server_negotiation.cc:407] Connection header received
0809 07:57:15.692771 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:15.692773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:15.692826 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:15.692908 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:57:15.693077 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.693559 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.694444 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:15.694652 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:15.694717 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:15.694932 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:57:15.695329 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:57:15.695412 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:15.695440 (+ 28us) server_negotiation.cc:299] Negotiation successful
0809 07:57:15.695484 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 07:57:15.701442 22995 raft_consensus.cc:1261] T 4edf402bf8914f30888ba2d4503d6165 P 34f56e1177fe4d4a820253f27b0defde [term 1 FOLLOWER]: Refusing update from remote peer 44c6ea408cea4954949ba0d343e72f8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:57:15.701442 23406 raft_consensus.cc:1261] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e [term 1 FOLLOWER]: Refusing update from remote peer 44c6ea408cea4954949ba0d343e72f8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:57:15.702162 23509 consensus_queue.cc:1035] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "34f56e1177fe4d4a820253f27b0defde" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 35577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:57:15.702368 23515 consensus_queue.cc:1035] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8325eed77daf48f7b8e31a875f3d809e" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 42169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:57:15.709941 23519 mvcc.cc:205] Tried to move back new op lower bound from 7316530117425455104 to 7316530117244858368. Current Snapshot: MvccSnapshot[applied={T|T < 7316530117425455104}]
I20260809 07:57:15.711015 23520 mvcc.cc:205] Tried to move back new op lower bound from 7316530117425455104 to 7316530117244858368. Current Snapshot: MvccSnapshot[applied={T|T < 7316530117425455104}]
I20260809 07:57:15.719748 22775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:57:15.719910 22775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:57:15.722132 22775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cc6240f1f7443db91d1aa474ad1cb57: Sending DeleteTablet for 3 replicas of tablet 4edf402bf8914f30888ba2d4503d6165
I20260809 07:57:15.722872 23182 tablet_service.cc:1514] Processing DeleteTablet for tablet 4edf402bf8914f30888ba2d4503d6165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:57:15 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:39738
I20260809 07:57:15.722980 22965 tablet_service.cc:1514] Processing DeleteTablet for tablet 4edf402bf8914f30888ba2d4503d6165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:57:15 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:56954
I20260809 07:57:15.723084 23382 tablet_service.cc:1514] Processing DeleteTablet for tablet 4edf402bf8914f30888ba2d4503d6165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:57:15 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:59126
I20260809 07:57:15.723469 23534 tablet_replica.cc:330] T 4edf402bf8914f30888ba2d4503d6165 P 34f56e1177fe4d4a820253f27b0defde: stopping tablet replica
I20260809 07:57:15.723542 23535 tablet_replica.cc:330] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e: stopping tablet replica
I20260809 07:57:15.723779 23534 raft_consensus.cc:2229] T 4edf402bf8914f30888ba2d4503d6165 P 34f56e1177fe4d4a820253f27b0defde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:57:15.723979 23533 tablet_replica.cc:330] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f: stopping tablet replica
I20260809 07:57:15.724098 23534 raft_consensus.cc:2258] T 4edf402bf8914f30888ba2d4503d6165 P 34f56e1177fe4d4a820253f27b0defde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:57:15.724285 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 22869
I20260809 07:57:15.724339 23533 raft_consensus.cc:2229] T 4edf402bf8914f30888ba2d4503d6165 P 44c6ea408cea4954949ba0d343e72f8f [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:57:15.746568 23535 raft_consensus.cc:2229] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:57:15.747100 23535 raft_consensus.cc:2258] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:57:15.748811 23535 ts_tablet_manager.cc:1905] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 07:57:15.750793 22755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.112.1:35577 (error 108)
I20260809 07:57:15.751034 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 23083
I20260809 07:57:15.753265 23535 ts_tablet_manager.cc:1918] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:57:15.753458 23535 log.cc:1196] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/4edf402bf8914f30888ba2d4503d6165
I20260809 07:57:15.753872 23535 ts_tablet_manager.cc:1939] T 4edf402bf8914f30888ba2d4503d6165 P 8325eed77daf48f7b8e31a875f3d809e: Deleting consensus metadata
I20260809 07:57:15.755292 22755 catalog_manager.cc:4841] TS 8325eed77daf48f7b8e31a875f3d809e (127.28.112.3:42169): tablet 4edf402bf8914f30888ba2d4503d6165 (table test-table [id=4c206129084447b3b0e2d5dab89bf3aa]) successfully deleted
I20260809 07:57:15.758127 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.757731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:35577 (local address 127.0.0.1:56964)
0809 07:57:15.757902 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:15.758038 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:35577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260809 07:57:15.758385 22755 catalog_manager.cc:4584] TS 34f56e1177fe4d4a820253f27b0defde (127.28.112.1:35577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4edf402bf8914f30888ba2d4503d6165: Network error: Client connection negotiation failed: client connection to 127.28.112.1:35577: connect: Connection refused (error 111)
I20260809 07:57:15.759147 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 23286
I20260809 07:57:15.766779 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:15.759188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:44243 (local address 127.0.0.1:39754)
0809 07:57:15.765904 (+ 6716us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:15.766697 (+ 793us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.2:44243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6612}
I20260809 07:57:15.780176 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 22730
2026-08-09T07:57:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2483 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-09T07:57:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:15Z Disabled control of system clock
I20260809 07:57:15.814646 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:45359
--webserver_interface=127.28.112.62
--webserver_port=0
--builtin_ntp_servers=127.28.112.20:34877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:45359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 07:57:15.925530 23543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:15.925763 23543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:15.925796 23543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:15.929617 23543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:15.929664 23543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:15.929678 23543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:15.929693 23543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:15.929708 23543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:15.934242 23543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:34877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:45359
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:45359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:15.935114 23543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:15.936327 23543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:15.944223 23551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:15.944278 23543 server_base.cc:1034] running on GCE node
W20260809 07:57:15.944231 23552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:15.944262 23554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:15.944944 23543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:15.945914 23543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:15.947106 23543 hybrid_clock.cc:648] HybridClock initialized: now 1786262235947083 us; error 35 us; skew 500 ppm
I20260809 07:57:15.948808 23543 webserver.cc:460] Webserver started at http://127.28.112.62:43095/ using document root <none> and password file <none>
I20260809 07:57:15.949338 23543 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:15.949404 23543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:15.949604 23543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:15.951550 23543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "0f62af55d27b4a21a3dcb727042ca0c4"
format_stamp: "Formatted at 2026-08-09 07:57:15 on dist-test-slave-vpvm"
server_key: "9ca58ad8db0e898048b2c91ce656eb47"
server_key_iv: "048877264f70ee4c81babab1c383a335"
server_key_version: "encryptionkey@0"
I20260809 07:57:15.952069 23543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "0f62af55d27b4a21a3dcb727042ca0c4"
format_stamp: "Formatted at 2026-08-09 07:57:15 on dist-test-slave-vpvm"
server_key: "9ca58ad8db0e898048b2c91ce656eb47"
server_key_iv: "048877264f70ee4c81babab1c383a335"
server_key_version: "encryptionkey@0"
I20260809 07:57:15.955439 23543 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 07:57:15.957552 23563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:15.958664 23543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:57:15.958775 23543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "0f62af55d27b4a21a3dcb727042ca0c4"
format_stamp: "Formatted at 2026-08-09 07:57:15 on dist-test-slave-vpvm"
server_key: "9ca58ad8db0e898048b2c91ce656eb47"
server_key_iv: "048877264f70ee4c81babab1c383a335"
server_key_version: "encryptionkey@0"
I20260809 07:57:15.958860 23543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:15.995880 23543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:15.996594 23543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:15.996767 23543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:16.003644 23543 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:45359
I20260809 07:57:16.003669 23652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:45359 every 8 connection(s)
I20260809 07:57:16.004676 23543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:16.007768 23653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:16.010440 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 23543
I20260809 07:57:16.010545 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:16.010803 29120 external_mini_cluster.cc:1428] Setting key b68fa0f2f124a3aa6298e336cc7cc16d
I20260809 07:57:16.013641 23653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4: Bootstrap starting.
I20260809 07:57:16.016007 23653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:16.016904 23653 log.cc:824] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:16.019285 23653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4: No bootstrap required, opened a new log
I20260809 07:57:16.021780 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:16.012202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37894 (local address 127.28.112.62:45359)
0809 07:57:16.012821 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:16.012835 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:16.012868 (+ 33us) server_negotiation.cc:407] Connection header received
0809 07:57:16.013461 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:16.013478 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:16.013873 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:16.014209 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:16.014640 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.016425 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.017704 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.018388 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.018557 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:16.019418 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:16.019424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:16.019449 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:16.019923 (+ 474us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:16.019994 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:16.020021 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:16.020115 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:16.020580 (+ 465us) server_negotiation.cc:299] Negotiation successful
0809 07:57:16.020869 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":129,"threads_started":1}
I20260809 07:57:16.023680 23653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f62af55d27b4a21a3dcb727042ca0c4" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45359 } }
I20260809 07:57:16.023904 23653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:16.023948 23653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f62af55d27b4a21a3dcb727042ca0c4, State: Initialized, Role: FOLLOWER
I20260809 07:57:16.024304 23653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f62af55d27b4a21a3dcb727042ca0c4" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45359 } }
I20260809 07:57:16.024410 23653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:16.024473 23653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:16.024564 23653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:16.025655 23653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f62af55d27b4a21a3dcb727042ca0c4" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45359 } }
I20260809 07:57:16.026041 23653 leader_election.cc:302] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f62af55d27b4a21a3dcb727042ca0c4; no voters:
I20260809 07:57:16.026371 23653 leader_election.cc:288] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:16.026460 23661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:16.026674 23661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4 [term 1 LEADER]: Becoming Leader. State: Replica: 0f62af55d27b4a21a3dcb727042ca0c4, State: Running, Role: LEADER
I20260809 07:57:16.026965 23661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f62af55d27b4a21a3dcb727042ca0c4" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45359 } }
I20260809 07:57:16.027565 23653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:16.028801 23662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f62af55d27b4a21a3dcb727042ca0c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f62af55d27b4a21a3dcb727042ca0c4" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45359 } } }
I20260809 07:57:16.029006 23662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:16.029354 23663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f62af55d27b4a21a3dcb727042ca0c4. Latest consensus state: current_term: 1 leader_uuid: "0f62af55d27b4a21a3dcb727042ca0c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f62af55d27b4a21a3dcb727042ca0c4" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 45359 } } }
I20260809 07:57:16.029517 23663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4 [sys.catalog]: This master's current role is: LEADER
W20260809 07:57:16.032123 23683 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:57:16.032202 23683 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:57:16.032265 23676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:16.033118 23676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:16.040408 23676 catalog_manager.cc:1347] Generated new cluster ID: fe93effaaf1a4557b20fce37e3ca12e5
I20260809 07:57:16.040472 23676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:16.065902 23676 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:16.067077 23676 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:16.075520 23676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4: Generated new TSK 0
I20260809 07:57:16.076248 23676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:16.090317 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:45359
--builtin_ntp_servers=127.28.112.20:34877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 07:57:16.198742 23688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:16.198961 23688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:16.198999 23688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:16.202883 23688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:16.202929 23688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:16.202992 23688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:57:16.207684 23688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:34877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:45359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:16.208623 23688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:16.209906 23688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:16.218915 23696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:16.218912 23695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:16.218935 23700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:16.219110 23688 server_base.cc:1034] running on GCE node
I20260809 07:57:16.219631 23688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:16.220273 23688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:16.221429 23688 hybrid_clock.cc:648] HybridClock initialized: now 1786262236221415 us; error 32 us; skew 500 ppm
I20260809 07:57:16.223205 23688 webserver.cc:460] Webserver started at http://127.28.112.1:44435/ using document root <none> and password file <none>
I20260809 07:57:16.223739 23688 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:16.223805 23688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:16.223996 23688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:16.225930 23688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "d125a2092c2742d68ccda616b46e4f84"
format_stamp: "Formatted at 2026-08-09 07:57:16 on dist-test-slave-vpvm"
server_key: "912be7a15f35ea6786592fe6087a79c2"
server_key_iv: "fb9d553a226c06cd32dacc10d453724b"
server_key_version: "encryptionkey@0"
I20260809 07:57:16.226518 23688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "d125a2092c2742d68ccda616b46e4f84"
format_stamp: "Formatted at 2026-08-09 07:57:16 on dist-test-slave-vpvm"
server_key: "912be7a15f35ea6786592fe6087a79c2"
server_key_iv: "fb9d553a226c06cd32dacc10d453724b"
server_key_version: "encryptionkey@0"
I20260809 07:57:16.230091 23688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 07:57:16.232326 23710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:16.233426 23688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:57:16.233534 23688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "d125a2092c2742d68ccda616b46e4f84"
format_stamp: "Formatted at 2026-08-09 07:57:16 on dist-test-slave-vpvm"
server_key: "912be7a15f35ea6786592fe6087a79c2"
server_key_iv: "fb9d553a226c06cd32dacc10d453724b"
server_key_version: "encryptionkey@0"
I20260809 07:57:16.233635 23688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:16.264817 23688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:16.265522 23688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:16.265698 23688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:16.266352 23688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:57:16.267333 23688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:57:16.267382 23688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:16.267441 23688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:57:16.267467 23688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:16.276048 23688 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:36379
I20260809 07:57:16.276115 23883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:36379 every 8 connection(s)
I20260809 07:57:16.276958 23688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:57:16.285732 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:16.278772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:47245 (local address 127.28.112.62:45359)
0809 07:57:16.278900 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:16.278903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:16.279901 (+ 998us) server_negotiation.cc:407] Connection header received
0809 07:57:16.281182 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:16.281185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:16.281231 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:16.281300 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:16.282136 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.282696 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.284126 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.284354 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.284441 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:16.285363 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:16.285366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:16.285368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:16.285381 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:16.285400 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:16.285407 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:16.285462 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:16.285562 (+ 100us) server_negotiation.cc:299] Negotiation successful
0809 07:57:16.285615 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 07:57:16.286527 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:16.279149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:45359 (local address 127.28.112.1:47245)
0809 07:57:16.279771 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:16.279810 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:16.281008 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:16.281445 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:16.281451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:16.281528 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:16.282025 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:16.282040 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.282801 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.282804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:16.283990 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:16.284002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.284478 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.284480 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:16.284630 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:16.285135 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:57:16.285137 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:16.285174 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:57:16.285232 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:16.285466 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:16.285471 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:16.285475 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 07:57:16.285557 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 07:57:16.285769 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":113,"threads_started":1}
I20260809 07:57:16.286967 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 23688
I20260809 07:57:16.287089 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:57:16.287364 29120 external_mini_cluster.cc:1428] Setting key bb01cd8b751fc04dac7305cc225053e8
I20260809 07:57:16.288302 23884 heartbeater.cc:347] Connected to a master server at 127.28.112.62:45359
I20260809 07:57:16.288519 23884 heartbeater.cc:464] Registering TS with master...
I20260809 07:57:16.289089 23884 heartbeater.cc:511] Master 127.28.112.62:45359 requested a full tablet report, sending...
I20260809 07:57:16.289238 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:45359
--builtin_ntp_servers=127.28.112.20:34877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 07:57:16.290643 23594 ts_manager.cc:194] Registered new tserver with Master: d125a2092c2742d68ccda616b46e4f84 (127.28.112.1:36379)
I20260809 07:57:16.291761 23594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.112.1:47245
W20260809 07:57:16.398757 23889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:16.398980 23889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:16.399015 23889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:16.402972 23889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:16.403017 23889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:16.403069 23889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:57:16.407749 23889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:34877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:45359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:16.408694 23889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:16.409986 23889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:16.419083 23895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:16.419094 23898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:16.419087 23896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:16.419296 23889 server_base.cc:1034] running on GCE node
I20260809 07:57:16.419687 23889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:16.420289 23889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:16.421442 23889 hybrid_clock.cc:648] HybridClock initialized: now 1786262236421426 us; error 27 us; skew 500 ppm
I20260809 07:57:16.423228 23889 webserver.cc:460] Webserver started at http://127.28.112.2:46779/ using document root <none> and password file <none>
I20260809 07:57:16.423771 23889 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:16.423831 23889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:16.424033 23889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:16.426081 23889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "796e60b5c391428aab214d28d33cc4a6"
format_stamp: "Formatted at 2026-08-09 07:57:16 on dist-test-slave-vpvm"
server_key: "fcf1b8abdf5f6db7eefa818509e98369"
server_key_iv: "1ae32d522d2b904f98660ab3f12b74d7"
server_key_version: "encryptionkey@0"
I20260809 07:57:16.426613 23889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "796e60b5c391428aab214d28d33cc4a6"
format_stamp: "Formatted at 2026-08-09 07:57:16 on dist-test-slave-vpvm"
server_key: "fcf1b8abdf5f6db7eefa818509e98369"
server_key_iv: "1ae32d522d2b904f98660ab3f12b74d7"
server_key_version: "encryptionkey@0"
I20260809 07:57:16.430235 23889 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 07:57:16.432492 23906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:16.433593 23889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:57:16.433696 23889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "796e60b5c391428aab214d28d33cc4a6"
format_stamp: "Formatted at 2026-08-09 07:57:16 on dist-test-slave-vpvm"
server_key: "fcf1b8abdf5f6db7eefa818509e98369"
server_key_iv: "1ae32d522d2b904f98660ab3f12b74d7"
server_key_version: "encryptionkey@0"
I20260809 07:57:16.433789 23889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:16.453790 23889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:16.454447 23889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:16.454609 23889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:16.455165 23889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:57:16.456099 23889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:57:16.456154 23889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:16.456205 23889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:57:16.456233 23889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:16.464520 23889 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:43429
I20260809 07:57:16.464543 24078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:43429 every 8 connection(s)
I20260809 07:57:16.465342 23889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:57:16.465834 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 23889
I20260809 07:57:16.465971 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:57:16.466265 29120 external_mini_cluster.cc:1428] Setting key d6db9281f575479dc4d0abaf23c3a943
I20260809 07:57:16.468214 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:45359
--builtin_ntp_servers=127.28.112.20:34877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 07:57:16.474620 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:16.467103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:51199 (local address 127.28.112.62:45359)
0809 07:57:16.467227 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:16.467232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:16.468396 (+ 1164us) server_negotiation.cc:407] Connection header received
0809 07:57:16.469725 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:16.469729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:16.469783 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:16.469854 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:16.470715 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.471235 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.472718 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.472937 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.473022 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:16.474196 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:16.474200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:16.474202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:16.474215 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:16.474236 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:16.474244 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:16.474300 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:16.474457 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 07:57:16.474509 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 07:57:16.475796 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:16.467561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:45359 (local address 127.28.112.2:51199)
0809 07:57:16.468260 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:16.468308 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:16.469525 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:16.470045 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:16.470052 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:16.470134 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:16.470573 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:16.470585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.471355 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.471360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:16.472600 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:16.472612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.473096 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.473101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:16.473299 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:16.473872 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:57:16.473874 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:16.473927 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:57:16.474042 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:16.474320 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:16.474328 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:16.474335 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 07:57:16.474464 (+ 129us) client_negotiation.cc:240] Negotiation successful
0809 07:57:16.474848 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":102,"threads_started":1}
I20260809 07:57:16.476879 24080 heartbeater.cc:347] Connected to a master server at 127.28.112.62:45359
I20260809 07:57:16.477120 24080 heartbeater.cc:464] Registering TS with master...
I20260809 07:57:16.477720 24080 heartbeater.cc:511] Master 127.28.112.62:45359 requested a full tablet report, sending...
I20260809 07:57:16.478829 23594 ts_manager.cc:194] Registered new tserver with Master: 796e60b5c391428aab214d28d33cc4a6 (127.28.112.2:43429)
I20260809 07:57:16.479377 23594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.112.2:51199
W20260809 07:57:16.580313 24085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:16.580549 24085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:16.580585 24085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:16.584493 24085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:16.584542 24085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:16.584604 24085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:57:16.589126 24085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:34877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:45359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:16.590061 24085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:16.591336 24085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:16.600064 24098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:16.600075 24092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:16.600075 24093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:16.600328 24085 server_base.cc:1034] running on GCE node
I20260809 07:57:16.600728 24085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:16.601289 24085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:16.602437 24085 hybrid_clock.cc:648] HybridClock initialized: now 1786262236602422 us; error 24 us; skew 500 ppm
I20260809 07:57:16.604067 24085 webserver.cc:460] Webserver started at http://127.28.112.3:39493/ using document root <none> and password file <none>
I20260809 07:57:16.604624 24085 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:16.604697 24085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:16.604878 24085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:16.606866 24085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "b81ff86532e94b4a9674e27f07f7d382"
format_stamp: "Formatted at 2026-08-09 07:57:16 on dist-test-slave-vpvm"
server_key: "4a76f5255c61cde6c7f2889119eb0881"
server_key_iv: "870911373801e2d123dee116b616dbe8"
server_key_version: "encryptionkey@0"
I20260809 07:57:16.607374 24085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "b81ff86532e94b4a9674e27f07f7d382"
format_stamp: "Formatted at 2026-08-09 07:57:16 on dist-test-slave-vpvm"
server_key: "4a76f5255c61cde6c7f2889119eb0881"
server_key_iv: "870911373801e2d123dee116b616dbe8"
server_key_version: "encryptionkey@0"
I20260809 07:57:16.610756 24085 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:57:16.612922 24106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:16.614085 24085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:57:16.614190 24085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "b81ff86532e94b4a9674e27f07f7d382"
format_stamp: "Formatted at 2026-08-09 07:57:16 on dist-test-slave-vpvm"
server_key: "4a76f5255c61cde6c7f2889119eb0881"
server_key_iv: "870911373801e2d123dee116b616dbe8"
server_key_version: "encryptionkey@0"
I20260809 07:57:16.614280 24085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:16.634761 24085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:16.635375 24085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:16.635537 24085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:16.636083 24085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:57:16.637032 24085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:57:16.637077 24085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:16.637130 24085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:57:16.637158 24085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:16.645684 24085 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:43157
I20260809 07:57:16.645709 24282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:43157 every 8 connection(s)
I20260809 07:57:16.646569 24085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:57:16.654779 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:16.648281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:36973 (local address 127.28.112.62:45359)
0809 07:57:16.648454 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:16.648459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:16.649431 (+ 972us) server_negotiation.cc:407] Connection header received
0809 07:57:16.650784 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:16.650787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:16.650834 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:16.650905 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:16.651697 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.652212 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.653226 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.653427 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.653490 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:16.654423 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:16.654426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:16.654428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:16.654440 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:16.654458 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:16.654466 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:16.654520 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:16.654619 (+ 99us) server_negotiation.cc:299] Negotiation successful
0809 07:57:16.654667 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 07:57:16.655395 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 24085
I20260809 07:57:16.655480 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:57:16.655568 24287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:16.648671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:45359 (local address 127.28.112.3:36973)
0809 07:57:16.649293 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:16.649331 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:16.650593 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:16.651051 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:16.651057 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:16.651134 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:16.651581 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:16.651592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.652298 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.652300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:16.653138 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:16.653147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.653529 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.653532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:16.653671 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:16.654193 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:57:16.654194 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:16.654231 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:57:16.654289 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:16.654526 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:16.654530 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:16.654533 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 07:57:16.654613 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 07:57:16.654821 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
I20260809 07:57:16.655757 29120 external_mini_cluster.cc:1428] Setting key 605cdf0f764be7ccedd8a2bb33c122ab
I20260809 07:57:16.656685 24283 heartbeater.cc:347] Connected to a master server at 127.28.112.62:45359
I20260809 07:57:16.656899 24283 heartbeater.cc:464] Registering TS with master...
I20260809 07:57:16.657447 24283 heartbeater.cc:511] Master 127.28.112.62:45359 requested a full tablet report, sending...
I20260809 07:57:16.658448 23594 ts_manager.cc:194] Registered new tserver with Master: b81ff86532e94b4a9674e27f07f7d382 (127.28.112.3:43157)
I20260809 07:57:16.658982 23594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.112.3:36973
I20260809 07:57:16.669097 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 07:57:16.674018 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:16.670921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37898 (local address 127.28.112.62:45359)
0809 07:57:16.671041 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:16.671046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:16.671195 (+ 149us) server_negotiation.cc:407] Connection header received
0809 07:57:16.671302 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:16.671304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:16.671342 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:16.671395 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:16.671613 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.672092 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.672816 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.673011 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.673076 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:16.673671 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:16.673675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:16.673676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:16.673688 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:16.673706 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:16.673714 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:16.673764 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:16.673822 (+ 58us) server_negotiation.cc:299] Negotiation successful
0809 07:57:16.673867 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 07:57:16.676358 23594 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:57:16.679332 23594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:57:16.693045 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:16.687889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40028 (local address 127.28.112.3:43157)
0809 07:57:16.688460 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:16.688471 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:16.688494 (+ 23us) server_negotiation.cc:407] Connection header received
0809 07:57:16.688560 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:16.688565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:16.688778 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:16.688925 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:16.689352 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.690574 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.691862 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.692397 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.692494 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:16.692685 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:16.692816 (+ 131us) server_negotiation.cc:299] Negotiation successful
0809 07:57:16.692906 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":160,"threads_started":1}
I20260809 07:57:16.693040 24300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:16.687245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:43157 (local address 127.0.0.1:40028)
0809 07:57:16.687855 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:16.687902 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:16.688039 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:16.689087 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:16.689093 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:16.689115 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:16.689245 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:16.689252 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.690732 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.690736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:16.691739 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:16.691750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.692508 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.692512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:16.692633 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:16.692672 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 07:57:16.692777 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 07:57:16.692891 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":34,"spinlock_wait_cycles":20736,"thread_start_us":90,"threads_started":1}
I20260809 07:57:16.693346 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:16.687483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:36379 (local address 127.0.0.1:40852)
0809 07:57:16.688468 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:16.688480 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:16.688572 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:16.689167 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:16.689170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:16.689188 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:16.689295 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:16.689302 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.690687 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.690691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:16.691807 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:16.691817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.692917 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.692921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:16.693032 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:16.693063 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 07:57:16.693132 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 07:57:16.693215 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"mutex_wait_us":11,"thread_start_us":100,"threads_started":1}
I20260809 07:57:16.693490 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:16.687260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34308 (local address 127.28.112.2:43429)
0809 07:57:16.688778 (+ 1518us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:16.688789 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:16.688812 (+ 23us) server_negotiation.cc:407] Connection header received
0809 07:57:16.688886 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:16.688891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:16.689060 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:16.689201 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:16.689517 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.690285 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.691749 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.692552 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.692662 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:16.692852 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:16.693251 (+ 399us) server_negotiation.cc:299] Negotiation successful
0809 07:57:16.693344 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1423,"thread_start_us":134,"threads_started":1}
I20260809 07:57:16.693779 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:16.687887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.28.112.1:36379)
0809 07:57:16.688356 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:16.688364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:16.688770 (+ 406us) server_negotiation.cc:407] Connection header received
0809 07:57:16.688853 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:16.688857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:16.689025 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:16.689182 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:16.689406 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.690574 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.691907 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.692619 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.692715 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:16.692851 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:16.693528 (+ 677us) server_negotiation.cc:299] Negotiation successful
0809 07:57:16.693630 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":144,"threads_started":1}
I20260809 07:57:16.694069 24303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:16.687116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:43429 (local address 127.0.0.1:34308)
0809 07:57:16.687855 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:16.687904 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:16.688039 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:16.689332 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:16.689335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:16.689351 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:16.689417 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:16.689421 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.690387 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.690391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:16.691631 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:16.691642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:16.692917 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:16.692921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:16.693055 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:16.693072 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 07:57:16.693144 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 07:57:16.693921 (+ 777us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":740,"spinlock_wait_cycles":5632,"thread_start_us":84,"threads_started":1}
I20260809 07:57:16.696286 24181 tablet_service.cc:1467] Processing CreateTablet for tablet c91233b57c344eaeaf630cedabf234e5 (DEFAULT_TABLE table=test-table [id=9644bfa7fda5454da80c17fb12b9baf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:57:16.696548 23984 tablet_service.cc:1467] Processing CreateTablet for tablet c91233b57c344eaeaf630cedabf234e5 (DEFAULT_TABLE table=test-table [id=9644bfa7fda5454da80c17fb12b9baf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:57:16.696636 23787 tablet_service.cc:1467] Processing CreateTablet for tablet c91233b57c344eaeaf630cedabf234e5 (DEFAULT_TABLE table=test-table [id=9644bfa7fda5454da80c17fb12b9baf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:57:16.697355 23984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91233b57c344eaeaf630cedabf234e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:16.697387 24181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91233b57c344eaeaf630cedabf234e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:16.697481 23787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91233b57c344eaeaf630cedabf234e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:16.703919 24310 tablet_bootstrap.cc:492] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6: Bootstrap starting.
I20260809 07:57:16.704768 24314 tablet_bootstrap.cc:492] T c91233b57c344eaeaf630cedabf234e5 P d125a2092c2742d68ccda616b46e4f84: Bootstrap starting.
I20260809 07:57:16.706272 24316 tablet_bootstrap.cc:492] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382: Bootstrap starting.
I20260809 07:57:16.706620 24310 tablet_bootstrap.cc:654] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:16.706852 24314 tablet_bootstrap.cc:654] T c91233b57c344eaeaf630cedabf234e5 P d125a2092c2742d68ccda616b46e4f84: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:16.707508 24310 log.cc:824] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:16.707533 24314 log.cc:824] T c91233b57c344eaeaf630cedabf234e5 P d125a2092c2742d68ccda616b46e4f84: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:16.707945 24316 tablet_bootstrap.cc:654] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:16.708722 24316 log.cc:824] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:16.709565 24310 tablet_bootstrap.cc:492] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6: No bootstrap required, opened a new log
I20260809 07:57:16.709565 24314 tablet_bootstrap.cc:492] T c91233b57c344eaeaf630cedabf234e5 P d125a2092c2742d68ccda616b46e4f84: No bootstrap required, opened a new log
I20260809 07:57:16.709760 24314 ts_tablet_manager.cc:1397] T c91233b57c344eaeaf630cedabf234e5 P d125a2092c2742d68ccda616b46e4f84: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 07:57:16.709760 24310 ts_tablet_manager.cc:1397] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 07:57:16.710433 24316 tablet_bootstrap.cc:492] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382: No bootstrap required, opened a new log
I20260809 07:57:16.710613 24316 ts_tablet_manager.cc:1397] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260809 07:57:16.713490 24316 raft_consensus.cc:348] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796e60b5c391428aab214d28d33cc4a6" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43429 } } peers { permanent_uuid: "b81ff86532e94b4a9674e27f07f7d382" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 43157 } } peers { permanent_uuid: "d125a2092c2742d68ccda616b46e4f84" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36379 } }
I20260809 07:57:16.713703 24316 raft_consensus.cc:374] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:16.713757 24316 raft_consensus.cc:729] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81ff86532e94b4a9674e27f07f7d382, State: Initialized, Role: FOLLOWER
I20260809 07:57:16.713944 24310 raft_consensus.cc:348] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796e60b5c391428aab214d28d33cc4a6" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43429 } } peers { permanent_uuid: "b81ff86532e94b4a9674e27f07f7d382" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 43157 } } peers { permanent_uuid: "d125a2092c2742d68ccda616b46e4f84" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36379 } }
I20260809 07:57:16.713944 24314 raft_consensus.cc:348] T c91233b57c344eaeaf630cedabf234e5 P d125a2092c2742d68ccda616b46e4f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796e60b5c391428aab214d28d33cc4a6" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43429 } } peers { permanent_uuid: "b81ff86532e94b4a9674e27f07f7d382" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 43157 } } peers { permanent_uuid: "d125a2092c2742d68ccda616b46e4f84" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36379 } }
I20260809 07:57:16.714227 24310 raft_consensus.cc:374] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:16.714239 24314 raft_consensus.cc:374] T c91233b57c344eaeaf630cedabf234e5 P d125a2092c2742d68ccda616b46e4f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:16.714183 24316 consensus_queue.cc:260] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796e60b5c391428aab214d28d33cc4a6" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43429 } } peers { permanent_uuid: "b81ff86532e94b4a9674e27f07f7d382" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 43157 } } peers { permanent_uuid: "d125a2092c2742d68ccda616b46e4f84" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36379 } }
I20260809 07:57:16.714288 24310 raft_consensus.cc:729] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796e60b5c391428aab214d28d33cc4a6, State: Initialized, Role: FOLLOWER
I20260809 07:57:16.714298 24314 raft_consensus.cc:729] T c91233b57c344eaeaf630cedabf234e5 P d125a2092c2742d68ccda616b46e4f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d125a2092c2742d68ccda616b46e4f84, State: Initialized, Role: FOLLOWER
I20260809 07:57:16.714795 24283 heartbeater.cc:503] Master 127.28.112.62:45359 was elected leader, sending a full tablet report...
I20260809 07:57:16.714743 24310 consensus_queue.cc:260] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796e60b5c391428aab214d28d33cc4a6" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43429 } } peers { permanent_uuid: "b81ff86532e94b4a9674e27f07f7d382" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 43157 } } peers { permanent_uuid: "d125a2092c2742d68ccda616b46e4f84" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36379 } }
I20260809 07:57:16.714743 24314 consensus_queue.cc:260] T c91233b57c344eaeaf630cedabf234e5 P d125a2092c2742d68ccda616b46e4f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796e60b5c391428aab214d28d33cc4a6" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43429 } } peers { permanent_uuid: "b81ff86532e94b4a9674e27f07f7d382" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 43157 } } peers { permanent_uuid: "d125a2092c2742d68ccda616b46e4f84" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36379 } }
I20260809 07:57:16.715149 24316 ts_tablet_manager.cc:1428] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260809 07:57:16.715649 24310 ts_tablet_manager.cc:1428] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260809 07:57:16.715639 24314 ts_tablet_manager.cc:1428] T c91233b57c344eaeaf630cedabf234e5 P d125a2092c2742d68ccda616b46e4f84: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 07:57:16.716028 24080 heartbeater.cc:503] Master 127.28.112.62:45359 was elected leader, sending a full tablet report...
I20260809 07:57:16.716120 23884 heartbeater.cc:503] Master 127.28.112.62:45359 was elected leader, sending a full tablet report...
W20260809 07:57:16.716346 24082 tablet.cc:2369] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 07:57:16.778047 23887 tablet.cc:2369] T c91233b57c344eaeaf630cedabf234e5 P d125a2092c2742d68ccda616b46e4f84: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 07:57:16.897532 24284 tablet.cc:2369] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:57:17.171839 24332 raft_consensus.cc:482] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:57:17.172096 24332 raft_consensus.cc:504] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796e60b5c391428aab214d28d33cc4a6" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43429 } } peers { permanent_uuid: "b81ff86532e94b4a9674e27f07f7d382" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 43157 } } peers { permanent_uuid: "d125a2092c2742d68ccda616b46e4f84" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36379 } }
I20260809 07:57:17.173318 24332 leader_election.cc:288] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b81ff86532e94b4a9674e27f07f7d382 (127.28.112.3:43157), d125a2092c2742d68ccda616b46e4f84 (127.28.112.1:36379)
I20260809 07:57:17.177047 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:17.173553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:43157 (local address 127.28.112.2:35739)
0809 07:57:17.173933 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:17.173995 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:17.174144 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:17.174409 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:17.174412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:17.174435 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:17.174554 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:17.174559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:17.175246 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:17.175250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:17.176050 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:17.176058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:17.176646 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:17.176650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:17.176731 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:17.176754 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 07:57:17.176816 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 07:57:17.176884 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":88,"threads_started":1}
I20260809 07:57:17.177026 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:17.173658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:35739 (local address 127.28.112.3:43157)
0809 07:57:17.173790 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:17.173795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:17.174097 (+ 302us) server_negotiation.cc:407] Connection header received
0809 07:57:17.174269 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:17.174273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:17.174324 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:17.174387 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:17.174652 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:17.175147 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:17.176165 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:17.176542 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:17.176646 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:17.176699 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:17.176838 (+ 139us) server_negotiation.cc:299] Negotiation successful
0809 07:57:17.176907 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 07:57:17.177253 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:17.173734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:44095 (local address 127.28.112.1:36379)
0809 07:57:17.173869 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:17.173873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:17.174148 (+ 275us) server_negotiation.cc:407] Connection header received
0809 07:57:17.174345 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:17.174349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:17.174420 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:17.174493 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:17.174790 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:17.175292 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:17.176314 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:17.176709 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:17.176827 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:17.176880 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:17.177064 (+ 184us) server_negotiation.cc:299] Negotiation successful
0809 07:57:17.177133 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 07:57:17.177269 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:17.173640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:36379 (local address 127.28.112.2:44095)
0809 07:57:17.174060 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:17.174073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:17.174182 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:17.174534 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:17.174538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:17.174557 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:17.174675 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:17.174681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:17.175404 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:17.175409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:17.176186 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:17.176196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:17.176834 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:17.176839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:17.176945 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:17.176965 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 07:57:17.177038 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 07:57:17.177107 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":95,"threads_started":1}
I20260809 07:57:17.177745 24212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91233b57c344eaeaf630cedabf234e5" candidate_uuid: "796e60b5c391428aab214d28d33cc4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81ff86532e94b4a9674e27f07f7d382" is_pre_election: true
I20260809 07:57:17.177824 23810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91233b57c344eaeaf630cedabf234e5" candidate_uuid: "796e60b5c391428aab214d28d33cc4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125a2092c2742d68ccda616b46e4f84" is_pre_election: true
I20260809 07:57:17.178105 24212 raft_consensus.cc:2454] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 796e60b5c391428aab214d28d33cc4a6 in term 0.
I20260809 07:57:17.178179 23810 raft_consensus.cc:2454] T c91233b57c344eaeaf630cedabf234e5 P d125a2092c2742d68ccda616b46e4f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 796e60b5c391428aab214d28d33cc4a6 in term 0.
I20260809 07:57:17.178622 23907 leader_election.cc:302] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 796e60b5c391428aab214d28d33cc4a6, b81ff86532e94b4a9674e27f07f7d382, d125a2092c2742d68ccda616b46e4f84; no voters:
I20260809 07:57:17.178874 24332 raft_consensus.cc:2781] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:57:17.178987 24332 raft_consensus.cc:482] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:57:17.179047 24332 raft_consensus.cc:3037] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:17.180135 24332 raft_consensus.cc:504] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796e60b5c391428aab214d28d33cc4a6" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43429 } } peers { permanent_uuid: "b81ff86532e94b4a9674e27f07f7d382" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 43157 } } peers { permanent_uuid: "d125a2092c2742d68ccda616b46e4f84" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36379 } }
I20260809 07:57:17.180496 24332 leader_election.cc:288] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [CANDIDATE]: Term 1 election: Requested vote from peers b81ff86532e94b4a9674e27f07f7d382 (127.28.112.3:43157), d125a2092c2742d68ccda616b46e4f84 (127.28.112.1:36379)
I20260809 07:57:17.180817 24212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91233b57c344eaeaf630cedabf234e5" candidate_uuid: "796e60b5c391428aab214d28d33cc4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81ff86532e94b4a9674e27f07f7d382"
I20260809 07:57:17.180835 23810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91233b57c344eaeaf630cedabf234e5" candidate_uuid: "796e60b5c391428aab214d28d33cc4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d125a2092c2742d68ccda616b46e4f84"
I20260809 07:57:17.180922 24212 raft_consensus.cc:3037] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:17.180946 23810 raft_consensus.cc:3037] T c91233b57c344eaeaf630cedabf234e5 P d125a2092c2742d68ccda616b46e4f84 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:17.181924 24212 raft_consensus.cc:2454] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 796e60b5c391428aab214d28d33cc4a6 in term 1.
I20260809 07:57:17.181939 23810 raft_consensus.cc:2454] T c91233b57c344eaeaf630cedabf234e5 P d125a2092c2742d68ccda616b46e4f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 796e60b5c391428aab214d28d33cc4a6 in term 1.
I20260809 07:57:17.182314 23910 leader_election.cc:302] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 796e60b5c391428aab214d28d33cc4a6, b81ff86532e94b4a9674e27f07f7d382, d125a2092c2742d68ccda616b46e4f84; no voters:
I20260809 07:57:17.182513 24332 raft_consensus.cc:2781] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:17.182731 24332 raft_consensus.cc:686] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [term 1 LEADER]: Becoming Leader. State: Replica: 796e60b5c391428aab214d28d33cc4a6, State: Running, Role: LEADER
I20260809 07:57:17.183053 24332 consensus_queue.cc:237] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796e60b5c391428aab214d28d33cc4a6" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43429 } } peers { permanent_uuid: "b81ff86532e94b4a9674e27f07f7d382" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 43157 } } peers { permanent_uuid: "d125a2092c2742d68ccda616b46e4f84" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36379 } }
I20260809 07:57:17.186378 23590 catalog_manager.cc:5495] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 796e60b5c391428aab214d28d33cc4a6 (127.28.112.2). New cstate: current_term: 1 leader_uuid: "796e60b5c391428aab214d28d33cc4a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796e60b5c391428aab214d28d33cc4a6" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 43429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b81ff86532e94b4a9674e27f07f7d382" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 43157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d125a2092c2742d68ccda616b46e4f84" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36379 } health_report { overall_health: UNKNOWN } } }
I20260809 07:57:17.254228 24348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:17.249186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.28.112.2:43429)
0809 07:57:17.249448 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:17.249453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:17.249467 (+ 14us) server_negotiation.cc:407] Connection header received
0809 07:57:17.249554 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:17.249557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:17.249635 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:17.249755 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:57:17.250473 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:17.251111 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:17.252309 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:17.252652 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:17.252747 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:17.253176 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:57:17.253735 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:57:17.253857 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:17.253992 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 07:57:17.254060 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":70,"threads_started":1}
I20260809 07:57:17.261667 23810 raft_consensus.cc:1261] T c91233b57c344eaeaf630cedabf234e5 P d125a2092c2742d68ccda616b46e4f84 [term 1 FOLLOWER]: Refusing update from remote peer 796e60b5c391428aab214d28d33cc4a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:57:17.261940 24212 raft_consensus.cc:1261] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382 [term 1 FOLLOWER]: Refusing update from remote peer 796e60b5c391428aab214d28d33cc4a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:57:17.262691 24332 consensus_queue.cc:1035] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d125a2092c2742d68ccda616b46e4f84" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:57:17.263134 24336 consensus_queue.cc:1035] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81ff86532e94b4a9674e27f07f7d382" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 43157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:57:17.272739 24351 mvcc.cc:205] Tried to move back new op lower bound from 7316530123815731200 to 7316530123504525312. Current Snapshot: MvccSnapshot[applied={T|T < 7316530123815731200}]
I20260809 07:57:17.273499 24355 mvcc.cc:205] Tried to move back new op lower bound from 7316530123815731200 to 7316530123504525312. Current Snapshot: MvccSnapshot[applied={T|T < 7316530123815731200}]
I20260809 07:57:17.274420 24358 mvcc.cc:205] Tried to move back new op lower bound from 7316530123815731200 to 7316530123504525312. Current Snapshot: MvccSnapshot[applied={T|T < 7316530123815731200}]
I20260809 07:57:17.285854 23591 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:57:17.286053 23591 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:57:17.288609 23591 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f62af55d27b4a21a3dcb727042ca0c4: Sending DeleteTablet for 3 replicas of tablet c91233b57c344eaeaf630cedabf234e5
I20260809 07:57:17.289374 23984 tablet_service.cc:1514] Processing DeleteTablet for tablet c91233b57c344eaeaf630cedabf234e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:57:17 UTC) from {username='slave'} at 127.0.0.1:34308
I20260809 07:57:17.289836 24181 tablet_service.cc:1514] Processing DeleteTablet for tablet c91233b57c344eaeaf630cedabf234e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:57:17 UTC) from {username='slave'} at 127.0.0.1:40028
I20260809 07:57:17.289826 23787 tablet_service.cc:1514] Processing DeleteTablet for tablet c91233b57c344eaeaf630cedabf234e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:57:17 UTC) from {username='slave'} at 127.0.0.1:40852
I20260809 07:57:17.290308 24369 tablet_replica.cc:330] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6: stopping tablet replica
I20260809 07:57:17.290326 24370 tablet_replica.cc:330] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382: stopping tablet replica
I20260809 07:57:17.290603 24369 raft_consensus.cc:2229] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:57:17.290634 24370 raft_consensus.cc:2229] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:57:17.290962 24370 raft_consensus.cc:2258] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:57:17.290975 24369 raft_consensus.cc:2258] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:57:17.291378 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 23688
I20260809 07:57:17.292259 24370 ts_tablet_manager.cc:1905] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:57:17.292299 24369 ts_tablet_manager.cc:1905] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:57:17.294916 24370 ts_tablet_manager.cc:1918] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:57:17.295011 24370 log.cc:1196] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/c91233b57c344eaeaf630cedabf234e5
I20260809 07:57:17.295128 24369 ts_tablet_manager.cc:1918] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:57:17.295212 24369 log.cc:1196] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/c91233b57c344eaeaf630cedabf234e5
I20260809 07:57:17.295349 24370 ts_tablet_manager.cc:1939] T c91233b57c344eaeaf630cedabf234e5 P b81ff86532e94b4a9674e27f07f7d382: Deleting consensus metadata
I20260809 07:57:17.295487 24369 ts_tablet_manager.cc:1939] T c91233b57c344eaeaf630cedabf234e5 P 796e60b5c391428aab214d28d33cc4a6: Deleting consensus metadata
I20260809 07:57:17.296590 23566 catalog_manager.cc:4841] TS b81ff86532e94b4a9674e27f07f7d382 (127.28.112.3:43157): tablet c91233b57c344eaeaf630cedabf234e5 (table test-table [id=9644bfa7fda5454da80c17fb12b9baf5]) successfully deleted
I20260809 07:57:17.296597 23571 catalog_manager.cc:4841] TS 796e60b5c391428aab214d28d33cc4a6 (127.28.112.2:43429): tablet c91233b57c344eaeaf630cedabf234e5 (table test-table [id=9644bfa7fda5454da80c17fb12b9baf5]) successfully deleted
W20260809 07:57:17.306226 23570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.112.1:36379 (error 108)
I20260809 07:57:17.306375 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 23889
I20260809 07:57:17.312644 24376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:17.312261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:36379 (local address 127.0.0.1:40854)
0809 07:57:17.312477 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:17.312542 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":60,"threads_started":1}
W20260809 07:57:17.312799 23570 catalog_manager.cc:4584] TS d125a2092c2742d68ccda616b46e4f84 (127.28.112.1:36379): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c91233b57c344eaeaf630cedabf234e5: Network error: Client connection negotiation failed: client connection to 127.28.112.1:36379: connect: Connection refused (error 111)
I20260809 07:57:17.316563 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 24085
I20260809 07:57:17.331895 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 23543
2026-08-09T07:57:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1549 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:17 dist-test-slave-vpvm krb5kdc[24379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:17 dist-test-slave-vpvm krb5kdc[24379](info): set up 2 sockets
Aug 09 07:57:17 dist-test-slave-vpvm krb5kdc[24379](info): commencing operation
krb5kdc: starting...
W20260809 07:57:18.462419 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:18 dist-test-slave-vpvm krb5kdc[24379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:57:18.578655 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:38417
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:38417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:18.685815 24394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:18.686100 24394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:18.686151 24394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:18.690001 24394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:18.690049 24394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:18.690068 24394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:18.690102 24394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:18.690131 24394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:18.695194 24394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:38417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:38417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:18.696136 24394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:18.697098 24394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:18.705052 24402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:18.705161 24394 server_base.cc:1034] running on GCE node
W20260809 07:57:18.705066 24403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:18.705052 24406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:18.705929 24394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:18.707048 24394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:18.708232 24394 hybrid_clock.cc:648] HybridClock initialized: now 1786262238708217 us; error 26 us; skew 500 ppm
Aug 09 07:57:18 dist-test-slave-vpvm krb5kdc[24379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262238, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:18.716907 24394 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:57:18.717746 24394 webserver.cc:460] Webserver started at http://127.28.112.62:42867/ using document root <none> and password file <none>
I20260809 07:57:18.718322 24394 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:18.718384 24394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:18.718550 24394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:18.720474 24394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "6d377947bfed4940ae3c66c217c8a7e8"
format_stamp: "Formatted at 2026-08-09 07:57:18 on dist-test-slave-vpvm"
server_key: "0bf1181bb53181ade4247651ed192f15"
server_key_iv: "9b3696b80430d32a3c94c1205b562685"
server_key_version: "encryptionkey@0"
I20260809 07:57:18.720973 24394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "6d377947bfed4940ae3c66c217c8a7e8"
format_stamp: "Formatted at 2026-08-09 07:57:18 on dist-test-slave-vpvm"
server_key: "0bf1181bb53181ade4247651ed192f15"
server_key_iv: "9b3696b80430d32a3c94c1205b562685"
server_key_version: "encryptionkey@0"
I20260809 07:57:18.724476 24394 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:57:18.726692 24415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:18.727821 24394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:57:18.727931 24394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "6d377947bfed4940ae3c66c217c8a7e8"
format_stamp: "Formatted at 2026-08-09 07:57:18 on dist-test-slave-vpvm"
server_key: "0bf1181bb53181ade4247651ed192f15"
server_key_iv: "9b3696b80430d32a3c94c1205b562685"
server_key_version: "encryptionkey@0"
I20260809 07:57:18.728029 24394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:18.741685 24394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:18.744590 24394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:18.744771 24394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:18.751351 24394 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:38417
I20260809 07:57:18.751358 24493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:38417 every 8 connection(s)
I20260809 07:57:18.752240 24394 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:18.753778 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 24394
I20260809 07:57:18.753909 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:18.754222 29120 external_mini_cluster.cc:1428] Setting key 21db32319f1bab87ce0e5c7bc733053f
I20260809 07:57:18.755163 24494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:18.761619 24494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8: Bootstrap starting.
Aug 09 07:57:18 dist-test-slave-vpvm krb5kdc[24379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:57:18.763788 24494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:18.764662 24494 log.cc:824] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:18.766727 24494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8: No bootstrap required, opened a new log
I20260809 07:57:18.768237 24499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:18.755519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39594 (local address 127.28.112.62:38417)
0809 07:57:18.755930 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:18.755954 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:18.755997 (+ 43us) server_negotiation.cc:407] Connection header received
0809 07:57:18.756788 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:18.756812 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:18.757212 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:18.757559 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:18.758236 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:18.759693 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:18.760507 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:18.761170 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:18.761331 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:18.763620 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:18.763626 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:18.763642 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:18.763672 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:18.765317 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:18.765713 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:18.765719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:18.765725 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:18.765856 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:18.766125 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:18.766128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:18.766130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:18.766658 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:18.766839 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:18.767105 (+ 266us) server_negotiation.cc:299] Negotiation successful
0809 07:57:18.767377 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":123,"threads_started":1}
I20260809 07:57:18.770962 24494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d377947bfed4940ae3c66c217c8a7e8" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38417 } }
I20260809 07:57:18.771157 24494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:18.771198 24494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d377947bfed4940ae3c66c217c8a7e8, State: Initialized, Role: FOLLOWER
I20260809 07:57:18.771524 24494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d377947bfed4940ae3c66c217c8a7e8" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38417 } }
I20260809 07:57:18.771641 24494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:18.771683 24494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:18.771750 24494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:18.772492 24494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d377947bfed4940ae3c66c217c8a7e8" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38417 } }
I20260809 07:57:18.772781 24494 leader_election.cc:302] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d377947bfed4940ae3c66c217c8a7e8; no voters:
I20260809 07:57:18.773039 24494 leader_election.cc:288] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:18.773145 24501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:18.773360 24501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8 [term 1 LEADER]: Becoming Leader. State: Replica: 6d377947bfed4940ae3c66c217c8a7e8, State: Running, Role: LEADER
I20260809 07:57:18.773661 24501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d377947bfed4940ae3c66c217c8a7e8" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38417 } }
I20260809 07:57:18.774233 24494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:18.775281 24502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d377947bfed4940ae3c66c217c8a7e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d377947bfed4940ae3c66c217c8a7e8" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38417 } } }
I20260809 07:57:18.775300 24503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d377947bfed4940ae3c66c217c8a7e8. Latest consensus state: current_term: 1 leader_uuid: "6d377947bfed4940ae3c66c217c8a7e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d377947bfed4940ae3c66c217c8a7e8" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38417 } } }
I20260809 07:57:18.775434 24502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:18.775437 24503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:18.775811 24516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:18.778332 24516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:18.783162 24516 catalog_manager.cc:1347] Generated new cluster ID: 7a7b3b8e76184e95ab9c9230ec2e4a65
I20260809 07:57:18.783226 24516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:18.799384 24516 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:18.800577 24516 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:18.811580 24516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8: Generated new TSK 0
I20260809 07:57:18.812321 24516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
I20260809 07:57:18.842381 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:38417
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:18.951244 24535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:18.951488 24535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:18.951545 24535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:18.955502 24535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:18.955551 24535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:18.955627 24535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:57:18.960606 24535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:18.961516 24535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:18.962484 24535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:18.971391 24546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:18.971422 24542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:18.971436 24541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:18.971557 24535 server_base.cc:1034] running on GCE node
I20260809 07:57:18.971916 24535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:18.972476 24535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:18.973625 24535 hybrid_clock.cc:648] HybridClock initialized: now 1786262238973608 us; error 30 us; skew 500 ppm
Aug 09 07:57:18 dist-test-slave-vpvm krb5kdc[24379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262238, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:18.976318 24535 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:57:18.977099 24535 webserver.cc:460] Webserver started at http://127.28.112.1:34743/ using document root <none> and password file <none>
I20260809 07:57:18.977618 24535 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:18.977676 24535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:18.977861 24535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:18.979780 24535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "ebdac2417b5a4f2ab112807476a506ed"
format_stamp: "Formatted at 2026-08-09 07:57:18 on dist-test-slave-vpvm"
server_key: "b4a615250a0dc19d5fcafa0790d5caa6"
server_key_iv: "ce0492198c18eaa065ac7a3431810f25"
server_key_version: "encryptionkey@0"
I20260809 07:57:18.980312 24535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "ebdac2417b5a4f2ab112807476a506ed"
format_stamp: "Formatted at 2026-08-09 07:57:18 on dist-test-slave-vpvm"
server_key: "b4a615250a0dc19d5fcafa0790d5caa6"
server_key_iv: "ce0492198c18eaa065ac7a3431810f25"
server_key_version: "encryptionkey@0"
I20260809 07:57:18.983739 24535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 07:57:18.985874 24561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:18.986972 24535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:57:18.987094 24535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "ebdac2417b5a4f2ab112807476a506ed"
format_stamp: "Formatted at 2026-08-09 07:57:18 on dist-test-slave-vpvm"
server_key: "b4a615250a0dc19d5fcafa0790d5caa6"
server_key_iv: "ce0492198c18eaa065ac7a3431810f25"
server_key_version: "encryptionkey@0"
I20260809 07:57:18.987181 24535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:19.005548 24535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:19.008109 24535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:19.008279 24535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:19.008863 24535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:57:19.009675 24535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:57:19.009723 24535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:19.009774 24535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:57:19.009802 24535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:19.018381 24535 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:32999
I20260809 07:57:19.018417 24722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:32999 every 8 connection(s)
I20260809 07:57:19.019275 24535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
Aug 09 07:57:19 dist-test-slave-vpvm krb5kdc[24379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262238, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:57:19.028330 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 24535
I20260809 07:57:19.028460 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:57:19.028801 29120 external_mini_cluster.cc:1428] Setting key 9e8c3f0f2027ebb775e0d02dbaffe08c
I20260809 07:57:19.032987 24499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:19.021018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:39229 (local address 127.28.112.62:38417)
0809 07:57:19.021169 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:19.021174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:19.022007 (+ 833us) server_negotiation.cc:407] Connection header received
0809 07:57:19.023237 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:19.023240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:19.023288 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:19.023359 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:19.024421 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.024943 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.025968 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.026209 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.026280 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:19.028451 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:19.028455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:19.028458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:19.028490 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:19.030956 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:19.031507 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:19.031509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:19.031511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:19.031583 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:19.031950 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:19.031953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:19.031955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:19.032134 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:19.032265 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:19.032705 (+ 440us) server_negotiation.cc:299] Negotiation successful
0809 07:57:19.032829 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 07:57:19.033670 24726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:19.021409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38417 (local address 127.28.112.1:39229)
0809 07:57:19.021840 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:19.021881 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:19.023034 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:19.023502 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:19.023508 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:19.023850 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:19.024309 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:19.024321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.025030 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.025032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:19.025859 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:19.025868 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.026326 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.026329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:19.026460 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:19.026937 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:19.026952 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:19.028280 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:19.031068 (+ 2788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:19.031074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:19.031089 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:19.031422 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:19.031670 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:19.031673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:19.031675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:19.031844 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:19.032232 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:19.032239 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:19.032491 (+ 252us) client_negotiation.cc:769] Sending connection context
0809 07:57:19.032647 (+ 156us) client_negotiation.cc:240] Negotiation successful
0809 07:57:19.032875 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":117,"threads_started":1}
I20260809 07:57:19.034792 24724 heartbeater.cc:347] Connected to a master server at 127.28.112.62:38417
I20260809 07:57:19.035009 24724 heartbeater.cc:464] Registering TS with master...
I20260809 07:57:19.035452 24724 heartbeater.cc:511] Master 127.28.112.62:38417 requested a full tablet report, sending...
I20260809 07:57:19.036875 24436 ts_manager.cc:194] Registered new tserver with Master: ebdac2417b5a4f2ab112807476a506ed (127.28.112.1:32999)
I20260809 07:57:19.037979 24436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:39229
WARNING: no policy specified for kudu/127.28.112.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
I20260809 07:57:19.054165 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:38417
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:19.165159 24730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:19.165392 24730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:19.165431 24730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:19.169477 24730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:19.169528 24730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:19.169581 24730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:57:19.174898 24730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:19.175854 24730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:19.176766 24730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:19.185793 24740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:19.185789 24737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:19.185817 24738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:19.186028 24730 server_base.cc:1034] running on GCE node
I20260809 07:57:19.186461 24730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:19.187040 24730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:19.188195 24730 hybrid_clock.cc:648] HybridClock initialized: now 1786262239188179 us; error 32 us; skew 500 ppm
Aug 09 07:57:19 dist-test-slave-vpvm krb5kdc[24379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262239, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:19.190773 24730 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:57:19.191541 24730 webserver.cc:460] Webserver started at http://127.28.112.2:45391/ using document root <none> and password file <none>
I20260809 07:57:19.192045 24730 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:19.192108 24730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:19.192284 24730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:19.194216 24730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "106801b5d48f4a49b9b08aa0be0c8ebf"
format_stamp: "Formatted at 2026-08-09 07:57:19 on dist-test-slave-vpvm"
server_key: "eeb6dde65a380c9e5627f9d3503b31ad"
server_key_iv: "54009af801d8d79d0abbfe5751537e3b"
server_key_version: "encryptionkey@0"
I20260809 07:57:19.194733 24730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "106801b5d48f4a49b9b08aa0be0c8ebf"
format_stamp: "Formatted at 2026-08-09 07:57:19 on dist-test-slave-vpvm"
server_key: "eeb6dde65a380c9e5627f9d3503b31ad"
server_key_iv: "54009af801d8d79d0abbfe5751537e3b"
server_key_version: "encryptionkey@0"
I20260809 07:57:19.198154 24730 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 07:57:19.200316 24750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:19.201380 24730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:57:19.201494 24730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "106801b5d48f4a49b9b08aa0be0c8ebf"
format_stamp: "Formatted at 2026-08-09 07:57:19 on dist-test-slave-vpvm"
server_key: "eeb6dde65a380c9e5627f9d3503b31ad"
server_key_iv: "54009af801d8d79d0abbfe5751537e3b"
server_key_version: "encryptionkey@0"
I20260809 07:57:19.201578 24730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:19.239840 24730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:19.242448 24730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:19.242640 24730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:19.243235 24730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:57:19.244061 24730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:57:19.244110 24730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:19.244165 24730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:57:19.244189 24730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:19.252532 24730 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:38399
I20260809 07:57:19.252558 24926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:38399 every 8 connection(s)
I20260809 07:57:19.253368 24730 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:57:19.260141 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 24730
I20260809 07:57:19.260269 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:57:19.260543 29120 external_mini_cluster.cc:1428] Setting key c49cf7cc701226b47c0dd3f97a111b87
Aug 09 07:57:19 dist-test-slave-vpvm krb5kdc[24379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262239, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:57:19.266639 24499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:19.255124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:52257 (local address 127.28.112.62:38417)
0809 07:57:19.255252 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:19.255255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:19.256027 (+ 772us) server_negotiation.cc:407] Connection header received
0809 07:57:19.257287 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:19.257291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:19.257352 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:19.257424 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:19.258570 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.259097 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.260206 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.260493 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.260583 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:19.263251 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:19.263256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:19.263259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:19.263288 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:19.264595 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:19.265157 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:19.265159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:19.265160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:19.265226 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:19.265547 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:19.265551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:19.265552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:19.265739 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:19.265827 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:19.266338 (+ 511us) server_negotiation.cc:299] Negotiation successful
0809 07:57:19.266463 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 07:57:19.267414 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:19.255452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38417 (local address 127.28.112.2:52257)
0809 07:57:19.255883 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:19.255923 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:19.257070 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:19.257600 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:19.257607 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:19.257937 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:19.258423 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:19.258436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.259220 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.259223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:19.260080 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:19.260090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.260646 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.260650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:19.260804 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:19.261359 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:19.261377 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:19.263056 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:19.264716 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:19.264722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:19.264736 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:19.265055 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:19.265297 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:19.265300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:19.265302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:19.265453 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:19.265840 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:19.265847 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:19.266114 (+ 267us) client_negotiation.cc:769] Sending connection context
0809 07:57:19.266279 (+ 165us) client_negotiation.cc:240] Negotiation successful
0809 07:57:19.266579 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":125,"threads_started":1}
I20260809 07:57:19.268530 24927 heartbeater.cc:347] Connected to a master server at 127.28.112.62:38417
I20260809 07:57:19.268795 24927 heartbeater.cc:464] Registering TS with master...
I20260809 07:57:19.269358 24927 heartbeater.cc:511] Master 127.28.112.62:38417 requested a full tablet report, sending...
I20260809 07:57:19.270529 24436 ts_manager.cc:194] Registered new tserver with Master: 106801b5d48f4a49b9b08aa0be0c8ebf (127.28.112.2:38399)
WARNING: no policy specified for kudu/127.28.112.3@KRBTEST.COM; defaulting to no policy
I20260809 07:57:19.271271 24436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:52257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
I20260809 07:57:19.287961 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:38417
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:19.398161 24938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:19.398394 24938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:19.398434 24938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:19.402307 24938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:19.402355 24938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:19.402408 24938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:57:19.407537 24938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:19.408457 24938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:19.409368 24938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:19.418365 24945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:19.418399 24944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:19.418511 24948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:19.418407 24938 server_base.cc:1034] running on GCE node
I20260809 07:57:19.418946 24938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:19.419534 24938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:19.420689 24938 hybrid_clock.cc:648] HybridClock initialized: now 1786262239420672 us; error 28 us; skew 500 ppm
Aug 09 07:57:19 dist-test-slave-vpvm krb5kdc[24379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262239, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:19.423319 24938 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:57:19.424129 24938 webserver.cc:460] Webserver started at http://127.28.112.3:45803/ using document root <none> and password file <none>
I20260809 07:57:19.424635 24938 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:19.424692 24938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:19.424856 24938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:19.426815 24938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "fb484bb9f2d1434cb7d7637158010dc2"
format_stamp: "Formatted at 2026-08-09 07:57:19 on dist-test-slave-vpvm"
server_key: "1b33a4ccc1c9cc9bf9b44ca01c04f477"
server_key_iv: "9ba59c49db1a82b1c3f489a51464b659"
server_key_version: "encryptionkey@0"
I20260809 07:57:19.427307 24938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "fb484bb9f2d1434cb7d7637158010dc2"
format_stamp: "Formatted at 2026-08-09 07:57:19 on dist-test-slave-vpvm"
server_key: "1b33a4ccc1c9cc9bf9b44ca01c04f477"
server_key_iv: "9ba59c49db1a82b1c3f489a51464b659"
server_key_version: "encryptionkey@0"
I20260809 07:57:19.430711 24938 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:57:19.432971 24957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:19.434116 24938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:57:19.434244 24938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "fb484bb9f2d1434cb7d7637158010dc2"
format_stamp: "Formatted at 2026-08-09 07:57:19 on dist-test-slave-vpvm"
server_key: "1b33a4ccc1c9cc9bf9b44ca01c04f477"
server_key_iv: "9ba59c49db1a82b1c3f489a51464b659"
server_key_version: "encryptionkey@0"
I20260809 07:57:19.434331 24938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:19.448736 24938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:19.451329 24938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:19.451490 24938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:19.452015 24938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:57:19.452811 24938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:57:19.452858 24938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:19.452909 24938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:57:19.452935 24938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:19.461463 24938 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:41559
I20260809 07:57:19.461480 25134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:41559 every 8 connection(s)
I20260809 07:57:19.462296 24938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:57:19.463474 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 24938
I20260809 07:57:19.463621 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:57:19.463959 29120 external_mini_cluster.cc:1428] Setting key 31198ee6ebe3e6b1d39e668a362ede5d
Aug 09 07:57:19 dist-test-slave-vpvm krb5kdc[24379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262239, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:57:19.474373 24499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:19.463971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:44785 (local address 127.28.112.62:38417)
0809 07:57:19.464101 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:19.464105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:19.465023 (+ 918us) server_negotiation.cc:407] Connection header received
0809 07:57:19.466305 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:19.466309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:19.466357 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:19.466420 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:19.467437 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.467964 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.469001 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.469225 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.469297 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:19.471529 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:19.471532 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:19.471534 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:19.471559 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:19.472714 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:19.473138 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:19.473140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:19.473141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:19.473206 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:19.473514 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:19.473516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:19.473517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:19.473677 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:19.473745 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:19.474169 (+ 424us) server_negotiation.cc:299] Negotiation successful
0809 07:57:19.474256 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 07:57:19.475203 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:19.464397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:38417 (local address 127.28.112.3:44785)
0809 07:57:19.464862 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:19.464912 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:19.466124 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:19.466560 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:19.466565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:19.466875 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:19.467328 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:19.467341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.468058 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.468060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:19.468885 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:19.468894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.469333 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.469335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:19.469473 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:19.469929 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:19.469945 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:19.471295 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:19.472809 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:19.472813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:19.472823 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:19.473068 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:19.473301 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:19.473304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:19.473305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:19.473440 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:19.473748 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:19.473752 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:19.473977 (+ 225us) client_negotiation.cc:769] Sending connection context
0809 07:57:19.474156 (+ 179us) client_negotiation.cc:240] Negotiation successful
0809 07:57:19.474389 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":129,"threads_started":1}
I20260809 07:57:19.476166 25136 heartbeater.cc:347] Connected to a master server at 127.28.112.62:38417
I20260809 07:57:19.476414 25136 heartbeater.cc:464] Registering TS with master...
I20260809 07:57:19.476902 25136 heartbeater.cc:511] Master 127.28.112.62:38417 requested a full tablet report, sending...
I20260809 07:57:19.477873 24436 ts_manager.cc:194] Registered new tserver with Master: fb484bb9f2d1434cb7d7637158010dc2 (127.28.112.3:41559)
I20260809 07:57:19.478417 24436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:44785
I20260809 07:57:19.488910 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 07:57:19.497748 24499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:19.490751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39600 (local address 127.28.112.62:38417)
0809 07:57:19.490862 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:19.490866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:19.491046 (+ 180us) server_negotiation.cc:407] Connection header received
0809 07:57:19.491159 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:19.491162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:19.491200 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:19.491256 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:19.491949 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.492432 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.493231 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.493435 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.493516 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:19.494943 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:19.494947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:19.494949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:19.494974 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:19.496101 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:19.496563 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:19.496566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:19.496567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:19.496638 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:19.496930 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:19.496934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:19.496935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:19.497133 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:19.497261 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:19.497521 (+ 260us) server_negotiation.cc:299] Negotiation successful
0809 07:57:19.497604 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 07:57:19.500303 24436 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:57:19.503214 24436 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:57:19.516942 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:19.511727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:38399 (local address 127.0.0.1:33806)
0809 07:57:19.512560 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:19.512573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:19.512661 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:19.513406 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:19.513409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:19.513423 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:19.513503 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:19.513509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.514509 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.514513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:19.515665 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:19.515675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.516435 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.516439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:19.516537 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:19.516578 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 07:57:19.516693 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 07:57:19.516799 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":88,"threads_started":1}
I20260809 07:57:19.516969 25155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:19.512097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.28.112.2:38399)
0809 07:57:19.512900 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:19.512908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:19.512927 (+ 19us) server_negotiation.cc:407] Connection header received
0809 07:57:19.512987 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:19.512991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:19.513124 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:19.513254 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:19.513593 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.514384 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.515782 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.516329 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.516430 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:19.516579 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:19.516734 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 07:57:19.516828 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":125,"threads_started":1}
I20260809 07:57:19.517676 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:19.512472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55842 (local address 127.28.112.1:32999)
0809 07:57:19.512749 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:19.512766 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:19.512789 (+ 23us) server_negotiation.cc:407] Connection header received
0809 07:57:19.512866 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:19.512872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:19.513074 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:19.513247 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:19.513531 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.514738 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.516351 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.517133 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.517229 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:19.517350 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:19.517449 (+ 99us) server_negotiation.cc:299] Negotiation successful
0809 07:57:19.517536 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":87,"threads_started":1}
I20260809 07:57:19.517794 25150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:19.511345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:41559 (local address 127.0.0.1:48346)
0809 07:57:19.512095 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:19.512154 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:19.512285 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:19.513415 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:19.513418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:19.513435 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:19.513538 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:19.513545 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.514958 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.514962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:19.516299 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:19.516308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.517414 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.517417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:19.517518 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:19.517540 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 07:57:19.517612 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 07:57:19.517674 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":82,"threads_started":1}
I20260809 07:57:19.517923 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:19.511477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.28.112.3:41559)
0809 07:57:19.512879 (+ 1402us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:19.512889 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:19.512907 (+ 18us) server_negotiation.cc:407] Connection header received
0809 07:57:19.512967 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:19.512971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:19.513108 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:19.513246 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:19.513680 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.514788 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.516422 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.517195 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.517297 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:19.517660 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:19.517709 (+ 49us) server_negotiation.cc:299] Negotiation successful
0809 07:57:19.517793 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1313,"thread_start_us":152,"threads_started":1}
I20260809 07:57:19.518379 25154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:19.511470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:32999 (local address 127.0.0.1:55842)
0809 07:57:19.512197 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:19.512210 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:19.512292 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:19.513257 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:19.513264 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:19.513286 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:19.513416 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:19.513423 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.515014 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.515017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:19.516204 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:19.516215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.517236 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.517240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:19.517328 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:19.517352 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 07:57:19.517425 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 07:57:19.518254 (+ 829us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":21,"thread_start_us":88,"threads_started":1}
I20260809 07:57:19.520450 25044 tablet_service.cc:1467] Processing CreateTablet for tablet 291e229ebdba438c9dba7c253f62392c (DEFAULT_TABLE table=test-table [id=d18a342f46fa47eab33070310e46b230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:57:19.520406 24826 tablet_service.cc:1467] Processing CreateTablet for tablet 291e229ebdba438c9dba7c253f62392c (DEFAULT_TABLE table=test-table [id=d18a342f46fa47eab33070310e46b230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:57:19.521384 25044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 291e229ebdba438c9dba7c253f62392c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:19.521446 24826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 291e229ebdba438c9dba7c253f62392c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:19.521078 24631 tablet_service.cc:1467] Processing CreateTablet for tablet 291e229ebdba438c9dba7c253f62392c (DEFAULT_TABLE table=test-table [id=d18a342f46fa47eab33070310e46b230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:57:19.522058 24631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 291e229ebdba438c9dba7c253f62392c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:19.527217 25158 tablet_bootstrap.cc:492] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2: Bootstrap starting.
I20260809 07:57:19.528946 25159 tablet_bootstrap.cc:492] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf: Bootstrap starting.
I20260809 07:57:19.528946 25161 tablet_bootstrap.cc:492] T 291e229ebdba438c9dba7c253f62392c P ebdac2417b5a4f2ab112807476a506ed: Bootstrap starting.
I20260809 07:57:19.529413 25158 tablet_bootstrap.cc:654] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:19.530467 25158 log.cc:824] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:19.531049 25161 tablet_bootstrap.cc:654] T 291e229ebdba438c9dba7c253f62392c P ebdac2417b5a4f2ab112807476a506ed: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:19.531438 25159 tablet_bootstrap.cc:654] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:19.532326 25159 log.cc:824] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:19.532327 25161 log.cc:824] T 291e229ebdba438c9dba7c253f62392c P ebdac2417b5a4f2ab112807476a506ed: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:19.532364 25158 tablet_bootstrap.cc:492] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2: No bootstrap required, opened a new log
I20260809 07:57:19.532552 25158 ts_tablet_manager.cc:1397] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 07:57:19.534312 25161 tablet_bootstrap.cc:492] T 291e229ebdba438c9dba7c253f62392c P ebdac2417b5a4f2ab112807476a506ed: No bootstrap required, opened a new log
I20260809 07:57:19.534322 25159 tablet_bootstrap.cc:492] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf: No bootstrap required, opened a new log
I20260809 07:57:19.534502 25159 ts_tablet_manager.cc:1397] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 07:57:19.534535 25161 ts_tablet_manager.cc:1397] T 291e229ebdba438c9dba7c253f62392c P ebdac2417b5a4f2ab112807476a506ed: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 07:57:19.535537 25158 raft_consensus.cc:348] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb484bb9f2d1434cb7d7637158010dc2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41559 } } peers { permanent_uuid: "ebdac2417b5a4f2ab112807476a506ed" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 32999 } } peers { permanent_uuid: "106801b5d48f4a49b9b08aa0be0c8ebf" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 38399 } }
I20260809 07:57:19.535766 25158 raft_consensus.cc:374] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:19.535838 25158 raft_consensus.cc:729] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb484bb9f2d1434cb7d7637158010dc2, State: Initialized, Role: FOLLOWER
I20260809 07:57:19.536249 25158 consensus_queue.cc:260] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb484bb9f2d1434cb7d7637158010dc2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41559 } } peers { permanent_uuid: "ebdac2417b5a4f2ab112807476a506ed" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 32999 } } peers { permanent_uuid: "106801b5d48f4a49b9b08aa0be0c8ebf" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 38399 } }
I20260809 07:57:19.537279 25158 ts_tablet_manager.cc:1428] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 07:57:19.537623 25136 heartbeater.cc:503] Master 127.28.112.62:38417 was elected leader, sending a full tablet report...
I20260809 07:57:19.538211 25161 raft_consensus.cc:348] T 291e229ebdba438c9dba7c253f62392c P ebdac2417b5a4f2ab112807476a506ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb484bb9f2d1434cb7d7637158010dc2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41559 } } peers { permanent_uuid: "ebdac2417b5a4f2ab112807476a506ed" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 32999 } } peers { permanent_uuid: "106801b5d48f4a49b9b08aa0be0c8ebf" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 38399 } }
I20260809 07:57:19.538211 25159 raft_consensus.cc:348] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb484bb9f2d1434cb7d7637158010dc2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41559 } } peers { permanent_uuid: "ebdac2417b5a4f2ab112807476a506ed" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 32999 } } peers { permanent_uuid: "106801b5d48f4a49b9b08aa0be0c8ebf" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 38399 } }
I20260809 07:57:19.538470 25159 raft_consensus.cc:374] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:19.538470 25161 raft_consensus.cc:374] T 291e229ebdba438c9dba7c253f62392c P ebdac2417b5a4f2ab112807476a506ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:19.538527 25161 raft_consensus.cc:729] T 291e229ebdba438c9dba7c253f62392c P ebdac2417b5a4f2ab112807476a506ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebdac2417b5a4f2ab112807476a506ed, State: Initialized, Role: FOLLOWER
I20260809 07:57:19.538564 25159 raft_consensus.cc:729] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106801b5d48f4a49b9b08aa0be0c8ebf, State: Initialized, Role: FOLLOWER
I20260809 07:57:19.539017 25161 consensus_queue.cc:260] T 291e229ebdba438c9dba7c253f62392c P ebdac2417b5a4f2ab112807476a506ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb484bb9f2d1434cb7d7637158010dc2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41559 } } peers { permanent_uuid: "ebdac2417b5a4f2ab112807476a506ed" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 32999 } } peers { permanent_uuid: "106801b5d48f4a49b9b08aa0be0c8ebf" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 38399 } }
I20260809 07:57:19.539017 25159 consensus_queue.cc:260] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb484bb9f2d1434cb7d7637158010dc2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41559 } } peers { permanent_uuid: "ebdac2417b5a4f2ab112807476a506ed" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 32999 } } peers { permanent_uuid: "106801b5d48f4a49b9b08aa0be0c8ebf" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 38399 } }
I20260809 07:57:19.539961 25159 ts_tablet_manager.cc:1428] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 07:57:19.539961 25161 ts_tablet_manager.cc:1428] T 291e229ebdba438c9dba7c253f62392c P ebdac2417b5a4f2ab112807476a506ed: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 07:57:19.540283 24927 heartbeater.cc:503] Master 127.28.112.62:38417 was elected leader, sending a full tablet report...
I20260809 07:57:19.540380 24724 heartbeater.cc:503] Master 127.28.112.62:38417 was elected leader, sending a full tablet report...
I20260809 07:57:19.685015 25170 raft_consensus.cc:482] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:57:19.685238 25170 raft_consensus.cc:504] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb484bb9f2d1434cb7d7637158010dc2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41559 } } peers { permanent_uuid: "ebdac2417b5a4f2ab112807476a506ed" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 32999 } } peers { permanent_uuid: "106801b5d48f4a49b9b08aa0be0c8ebf" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 38399 } }
I20260809 07:57:19.686430 25170 leader_election.cc:288] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb484bb9f2d1434cb7d7637158010dc2 (127.28.112.3:41559), ebdac2417b5a4f2ab112807476a506ed (127.28.112.1:32999)
I20260809 07:57:19.689920 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:19.686686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:41559 (local address 127.28.112.2:49037)
0809 07:57:19.686798 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:19.686813 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:19.686913 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:19.687233 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:19.687237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:19.687286 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:19.687384 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:19.687389 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.688094 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.688098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:19.688888 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:19.688898 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.689525 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.689529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:19.689630 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:19.689651 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 07:57:19.689716 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 07:57:19.689782 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"spinlock_wait_cycles":9216}
I20260809 07:57:19.689920 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:19.686921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:49037 (local address 127.28.112.3:41559)
0809 07:57:19.687041 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:19.687044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:19.687055 (+ 11us) server_negotiation.cc:407] Connection header received
0809 07:57:19.687101 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:19.687103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:19.687140 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:19.687205 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:19.687494 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.687986 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.689002 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.689391 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.689525 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:19.689581 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:19.689740 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 07:57:19.689803 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 07:57:19.690297 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:19.686821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:52321 (local address 127.28.112.1:32999)
0809 07:57:19.686955 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:19.686959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:19.687256 (+ 297us) server_negotiation.cc:407] Connection header received
0809 07:57:19.687396 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:19.687399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:19.687444 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:19.687509 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:19.687768 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.688253 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.689232 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.689647 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.689752 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:19.689803 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:19.690066 (+ 263us) server_negotiation.cc:299] Negotiation successful
0809 07:57:19.690173 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 07:57:19.690675 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:19.686730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:32999 (local address 127.28.112.2:52321)
0809 07:57:19.687149 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:19.687161 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:19.687291 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:19.687526 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:19.687529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:19.687544 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:19.687661 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:19.687666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.688356 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.688360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:19.689099 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:19.689107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.689752 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.689756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:19.689850 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:19.689874 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 07:57:19.690433 (+ 559us) client_negotiation.cc:240] Negotiation successful
0809 07:57:19.690507 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"spinlock_wait_cycles":29952,"thread_start_us":142,"threads_started":1}
I20260809 07:57:19.690740 25070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "291e229ebdba438c9dba7c253f62392c" candidate_uuid: "106801b5d48f4a49b9b08aa0be0c8ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb484bb9f2d1434cb7d7637158010dc2" is_pre_election: true
I20260809 07:57:19.691032 25070 raft_consensus.cc:2454] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 106801b5d48f4a49b9b08aa0be0c8ebf in term 0.
I20260809 07:57:19.691237 24661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "291e229ebdba438c9dba7c253f62392c" candidate_uuid: "106801b5d48f4a49b9b08aa0be0c8ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebdac2417b5a4f2ab112807476a506ed" is_pre_election: true
I20260809 07:57:19.691499 24756 leader_election.cc:302] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106801b5d48f4a49b9b08aa0be0c8ebf, fb484bb9f2d1434cb7d7637158010dc2; no voters:
I20260809 07:57:19.691540 24661 raft_consensus.cc:2454] T 291e229ebdba438c9dba7c253f62392c P ebdac2417b5a4f2ab112807476a506ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 106801b5d48f4a49b9b08aa0be0c8ebf in term 0.
I20260809 07:57:19.691763 25170 raft_consensus.cc:2781] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:57:19.691847 25170 raft_consensus.cc:482] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:57:19.691898 25170 raft_consensus.cc:3037] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:19.692932 25170 raft_consensus.cc:504] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb484bb9f2d1434cb7d7637158010dc2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41559 } } peers { permanent_uuid: "ebdac2417b5a4f2ab112807476a506ed" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 32999 } } peers { permanent_uuid: "106801b5d48f4a49b9b08aa0be0c8ebf" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 38399 } }
I20260809 07:57:19.693274 25170 leader_election.cc:288] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [CANDIDATE]: Term 1 election: Requested vote from peers fb484bb9f2d1434cb7d7637158010dc2 (127.28.112.3:41559), ebdac2417b5a4f2ab112807476a506ed (127.28.112.1:32999)
I20260809 07:57:19.693619 25070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "291e229ebdba438c9dba7c253f62392c" candidate_uuid: "106801b5d48f4a49b9b08aa0be0c8ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb484bb9f2d1434cb7d7637158010dc2"
I20260809 07:57:19.693619 24661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "291e229ebdba438c9dba7c253f62392c" candidate_uuid: "106801b5d48f4a49b9b08aa0be0c8ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebdac2417b5a4f2ab112807476a506ed"
I20260809 07:57:19.693725 25070 raft_consensus.cc:3037] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:19.693734 24661 raft_consensus.cc:3037] T 291e229ebdba438c9dba7c253f62392c P ebdac2417b5a4f2ab112807476a506ed [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:19.694743 25070 raft_consensus.cc:2454] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 106801b5d48f4a49b9b08aa0be0c8ebf in term 1.
I20260809 07:57:19.694816 24661 raft_consensus.cc:2454] T 291e229ebdba438c9dba7c253f62392c P ebdac2417b5a4f2ab112807476a506ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 106801b5d48f4a49b9b08aa0be0c8ebf in term 1.
I20260809 07:57:19.695013 24756 leader_election.cc:302] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106801b5d48f4a49b9b08aa0be0c8ebf, fb484bb9f2d1434cb7d7637158010dc2; no voters:
I20260809 07:57:19.695191 25170 raft_consensus.cc:2781] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:19.695410 25170 raft_consensus.cc:686] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [term 1 LEADER]: Becoming Leader. State: Replica: 106801b5d48f4a49b9b08aa0be0c8ebf, State: Running, Role: LEADER
I20260809 07:57:19.695727 25170 consensus_queue.cc:237] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb484bb9f2d1434cb7d7637158010dc2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41559 } } peers { permanent_uuid: "ebdac2417b5a4f2ab112807476a506ed" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 32999 } } peers { permanent_uuid: "106801b5d48f4a49b9b08aa0be0c8ebf" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 38399 } }
I20260809 07:57:19.698978 24433 catalog_manager.cc:5495] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf reported cstate change: term changed from 0 to 1, leader changed from <none> to 106801b5d48f4a49b9b08aa0be0c8ebf (127.28.112.2). New cstate: current_term: 1 leader_uuid: "106801b5d48f4a49b9b08aa0be0c8ebf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb484bb9f2d1434cb7d7637158010dc2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebdac2417b5a4f2ab112807476a506ed" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 32999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "106801b5d48f4a49b9b08aa0be0c8ebf" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 38399 } health_report { overall_health: HEALTHY } } }
W20260809 07:57:19.713156 25139 tablet.cc:2369] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 07:57:19.754441 24928 tablet.cc:2369] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:57:19.755647 25155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:19.751789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33808 (local address 127.28.112.2:38399)
0809 07:57:19.751930 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:19.751935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:19.751950 (+ 15us) server_negotiation.cc:407] Connection header received
0809 07:57:19.752029 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:19.752032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:19.752101 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:19.752206 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:57:19.752422 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.753209 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.753986 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:19.754282 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:19.754388 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:19.754788 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:57:19.755206 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:57:19.755320 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:19.755467 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 07:57:19.755502 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 07:57:19.761648 24661 raft_consensus.cc:1261] T 291e229ebdba438c9dba7c253f62392c P ebdac2417b5a4f2ab112807476a506ed [term 1 FOLLOWER]: Refusing update from remote peer 106801b5d48f4a49b9b08aa0be0c8ebf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:57:19.761648 25070 raft_consensus.cc:1261] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2 [term 1 FOLLOWER]: Refusing update from remote peer 106801b5d48f4a49b9b08aa0be0c8ebf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:57:19.762446 25173 consensus_queue.cc:1035] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebdac2417b5a4f2ab112807476a506ed" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 32999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:57:19.762620 25170 consensus_queue.cc:1035] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb484bb9f2d1434cb7d7637158010dc2" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 41559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:57:19.770147 25178 mvcc.cc:205] Tried to move back new op lower bound from 7316530134056185856 to 7316530133796196352. Current Snapshot: MvccSnapshot[applied={T|T < 7316530134056185856}]
I20260809 07:57:19.778820 24433 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:57:19.778972 24433 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:57:19.781157 24433 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d377947bfed4940ae3c66c217c8a7e8: Sending DeleteTablet for 3 replicas of tablet 291e229ebdba438c9dba7c253f62392c
I20260809 07:57:19.781858 24631 tablet_service.cc:1514] Processing DeleteTablet for tablet 291e229ebdba438c9dba7c253f62392c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:57:19 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:55842
I20260809 07:57:19.781858 24826 tablet_service.cc:1514] Processing DeleteTablet for tablet 291e229ebdba438c9dba7c253f62392c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:57:19 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:33806
I20260809 07:57:19.782382 25192 tablet_replica.cc:330] T 291e229ebdba438c9dba7c253f62392c P ebdac2417b5a4f2ab112807476a506ed: stopping tablet replica
I20260809 07:57:19.782387 25044 tablet_service.cc:1514] Processing DeleteTablet for tablet 291e229ebdba438c9dba7c253f62392c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:57:19 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:48346
I20260809 07:57:19.782627 25191 tablet_replica.cc:330] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf: stopping tablet replica
I20260809 07:57:19.782920 25191 raft_consensus.cc:2229] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:57:19.782944 25193 tablet_replica.cc:330] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2: stopping tablet replica
I20260809 07:57:19.783308 25191 raft_consensus.cc:2258] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:57:19.783310 25193 raft_consensus.cc:2229] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:57:19.783460 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 24535
I20260809 07:57:19.783746 25192 raft_consensus.cc:2229] T 291e229ebdba438c9dba7c253f62392c P ebdac2417b5a4f2ab112807476a506ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:57:19.783855 25193 raft_consensus.cc:2258] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:57:19.788412 25193 ts_tablet_manager.cc:1905] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:57:19.792141 25191 ts_tablet_manager.cc:1905] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:57:19.794652 25191 ts_tablet_manager.cc:1918] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:57:19.794750 25191 log.cc:1196] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/291e229ebdba438c9dba7c253f62392c
I20260809 07:57:19.795063 25191 ts_tablet_manager.cc:1939] T 291e229ebdba438c9dba7c253f62392c P 106801b5d48f4a49b9b08aa0be0c8ebf: Deleting consensus metadata
I20260809 07:57:19.796247 25193 ts_tablet_manager.cc:1918] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:57:19.796278 24416 catalog_manager.cc:4841] TS 106801b5d48f4a49b9b08aa0be0c8ebf (127.28.112.2:38399): tablet 291e229ebdba438c9dba7c253f62392c (table test-table [id=d18a342f46fa47eab33070310e46b230]) successfully deleted
I20260809 07:57:19.796370 25193 log.cc:1196] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/291e229ebdba438c9dba7c253f62392c
I20260809 07:57:19.796715 25193 ts_tablet_manager.cc:1939] T 291e229ebdba438c9dba7c253f62392c P fb484bb9f2d1434cb7d7637158010dc2: Deleting consensus metadata
I20260809 07:57:19.797835 24419 catalog_manager.cc:4841] TS fb484bb9f2d1434cb7d7637158010dc2 (127.28.112.3:41559): tablet 291e229ebdba438c9dba7c253f62392c (table test-table [id=d18a342f46fa47eab33070310e46b230]) successfully deleted
W20260809 07:57:19.800380 24420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.112.1:32999 (error 108)
I20260809 07:57:19.800616 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 24730
I20260809 07:57:19.810788 25154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:19.804950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:32999 (local address 127.0.0.1:55852)
0809 07:57:19.810607 (+ 5657us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:19.810687 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5558}
W20260809 07:57:19.811028 24420 catalog_manager.cc:4584] TS ebdac2417b5a4f2ab112807476a506ed (127.28.112.1:32999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 291e229ebdba438c9dba7c253f62392c: Network error: Client connection negotiation failed: client connection to 127.28.112.1:32999: connect: Connection refused (error 111)
I20260809 07:57:19.817690 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 24938
I20260809 07:57:19.826508 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 24394
2026-08-09T07:57:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2499 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-09T07:57:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:19Z Disabled control of system clock
I20260809 07:57:19.862337 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:37097
--webserver_interface=127.28.112.62
--webserver_port=0
--builtin_ntp_servers=127.28.112.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:37097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 07:57:19.970822 25208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:19.971064 25208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:19.971099 25208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:19.975088 25208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:19.975137 25208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:19.975152 25208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:19.975178 25208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:19.975205 25208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:19.979878 25208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:37097
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:37097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:19.980847 25208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:19.982199 25208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:19.990365 25214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:19.990576 25208 server_base.cc:1034] running on GCE node
W20260809 07:57:19.990402 25220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:19.990428 25216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:19.991168 25208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:19.992223 25208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:19.993374 25208 hybrid_clock.cc:648] HybridClock initialized: now 1786262239993357 us; error 33 us; skew 500 ppm
I20260809 07:57:19.995124 25208 webserver.cc:460] Webserver started at http://127.28.112.62:38303/ using document root <none> and password file <none>
I20260809 07:57:19.995678 25208 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:19.995746 25208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:19.995954 25208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:19.997924 25208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "33a487e3cc8140b19d804c1106d96cdb"
format_stamp: "Formatted at 2026-08-09 07:57:19 on dist-test-slave-vpvm"
server_key: "4e13410dee4489b14022b488b78c02e3"
server_key_iv: "141ea14d7b5dcb09f941b9631222979c"
server_key_version: "encryptionkey@0"
I20260809 07:57:19.998497 25208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "33a487e3cc8140b19d804c1106d96cdb"
format_stamp: "Formatted at 2026-08-09 07:57:19 on dist-test-slave-vpvm"
server_key: "4e13410dee4489b14022b488b78c02e3"
server_key_iv: "141ea14d7b5dcb09f941b9631222979c"
server_key_version: "encryptionkey@0"
I20260809 07:57:20.002024 25208 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 07:57:20.004290 25227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:20.005450 25208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 07:57:20.005564 25208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "33a487e3cc8140b19d804c1106d96cdb"
format_stamp: "Formatted at 2026-08-09 07:57:19 on dist-test-slave-vpvm"
server_key: "4e13410dee4489b14022b488b78c02e3"
server_key_iv: "141ea14d7b5dcb09f941b9631222979c"
server_key_version: "encryptionkey@0"
I20260809 07:57:20.005663 25208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:20.041625 25208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:20.042418 25208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:20.042610 25208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:20.049305 25208 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:37097
I20260809 07:57:20.049326 25301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:37097 every 8 connection(s)
I20260809 07:57:20.050292 25208 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:20.053369 25302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:20.057857 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 25208
I20260809 07:57:20.058001 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:20.058269 29120 external_mini_cluster.cc:1428] Setting key 64396b27c46ea39b6a089ea29da628c9
I20260809 07:57:20.059115 25302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb: Bootstrap starting.
I20260809 07:57:20.061745 25302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:20.062673 25302 log.cc:824] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:20.064678 25302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb: No bootstrap required, opened a new log
I20260809 07:57:20.068426 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.059657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.28.112.62:37097)
0809 07:57:20.060352 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:20.060372 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:20.060416 (+ 44us) server_negotiation.cc:407] Connection header received
0809 07:57:20.061221 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:20.061246 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:20.061754 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:20.062147 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:20.062595 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.063922 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.064809 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.065466 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.065642 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:20.066376 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:20.066382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:20.066400 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:20.066897 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:20.066968 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:20.066994 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:20.067093 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:20.067183 (+ 90us) server_negotiation.cc:299] Negotiation successful
0809 07:57:20.067450 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":99,"threads_started":1}
I20260809 07:57:20.068981 25302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a487e3cc8140b19d804c1106d96cdb" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37097 } }
I20260809 07:57:20.069199 25302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:20.069237 25302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a487e3cc8140b19d804c1106d96cdb, State: Initialized, Role: FOLLOWER
I20260809 07:57:20.069654 25302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a487e3cc8140b19d804c1106d96cdb" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37097 } }
I20260809 07:57:20.069789 25302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:20.069852 25302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:20.069972 25302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:20.070886 25302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a487e3cc8140b19d804c1106d96cdb" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37097 } }
I20260809 07:57:20.071210 25302 leader_election.cc:302] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33a487e3cc8140b19d804c1106d96cdb; no voters:
I20260809 07:57:20.071475 25302 leader_election.cc:288] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:20.071585 25307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:20.071780 25307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb [term 1 LEADER]: Becoming Leader. State: Replica: 33a487e3cc8140b19d804c1106d96cdb, State: Running, Role: LEADER
I20260809 07:57:20.072065 25307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a487e3cc8140b19d804c1106d96cdb" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37097 } }
I20260809 07:57:20.072585 25302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:20.073535 25308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33a487e3cc8140b19d804c1106d96cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a487e3cc8140b19d804c1106d96cdb" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37097 } } }
I20260809 07:57:20.073675 25308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:20.073940 25311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33a487e3cc8140b19d804c1106d96cdb. Latest consensus state: current_term: 1 leader_uuid: "33a487e3cc8140b19d804c1106d96cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a487e3cc8140b19d804c1106d96cdb" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 37097 } } }
I20260809 07:57:20.074066 25311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:20.074081 25321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:20.077117 25321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:20.081970 25321 catalog_manager.cc:1347] Generated new cluster ID: ab8ebf1aec424ee9a9aee6170e2968e9
I20260809 07:57:20.082027 25321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:20.112071 25321 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:20.113284 25321 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:20.122313 25321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb: Generated new TSK 0
I20260809 07:57:20.122942 25321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:20.133647 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:37097
--builtin_ntp_servers=127.28.112.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 07:57:20.241552 25338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:20.241784 25338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:20.241822 25338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:20.245714 25338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:20.245769 25338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:20.245832 25338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:57:20.250548 25338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:37097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:20.251586 25338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:20.253113 25338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:20.263193 25344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:20.263242 25338 server_base.cc:1034] running on GCE node
W20260809 07:57:20.263195 25345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:20.263195 25347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:20.263784 25338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:20.264369 25338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:20.265509 25338 hybrid_clock.cc:648] HybridClock initialized: now 1786262240265480 us; error 43 us; skew 500 ppm
I20260809 07:57:20.267206 25338 webserver.cc:460] Webserver started at http://127.28.112.1:41605/ using document root <none> and password file <none>
I20260809 07:57:20.267736 25338 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:20.267797 25338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:20.267998 25338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:20.269994 25338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "b6c6a4930d4845eab1ef6478700a5281"
format_stamp: "Formatted at 2026-08-09 07:57:20 on dist-test-slave-vpvm"
server_key: "1c5e666ff3b663e953c742eca7adc623"
server_key_iv: "4620552935cf66a18e9ffae3e85c1f71"
server_key_version: "encryptionkey@0"
I20260809 07:57:20.270538 25338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "b6c6a4930d4845eab1ef6478700a5281"
format_stamp: "Formatted at 2026-08-09 07:57:20 on dist-test-slave-vpvm"
server_key: "1c5e666ff3b663e953c742eca7adc623"
server_key_iv: "4620552935cf66a18e9ffae3e85c1f71"
server_key_version: "encryptionkey@0"
I20260809 07:57:20.274020 25338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 07:57:20.276305 25359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:20.277390 25338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:57:20.277488 25338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "b6c6a4930d4845eab1ef6478700a5281"
format_stamp: "Formatted at 2026-08-09 07:57:20 on dist-test-slave-vpvm"
server_key: "1c5e666ff3b663e953c742eca7adc623"
server_key_iv: "4620552935cf66a18e9ffae3e85c1f71"
server_key_version: "encryptionkey@0"
I20260809 07:57:20.277583 25338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:20.291818 25338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:20.292428 25338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:20.292590 25338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:20.293139 25338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:57:20.294080 25338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:57:20.294128 25338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:20.294191 25338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:57:20.294219 25338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:20.302501 25338 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:41375
I20260809 07:57:20.302526 25538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:41375 every 8 connection(s)
I20260809 07:57:20.303375 25338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:57:20.310148 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 25338
I20260809 07:57:20.310277 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:57:20.310606 29120 external_mini_cluster.cc:1428] Setting key 36744c45d99c49c379ed68c68d87ec09
I20260809 07:57:20.312482 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:37097
--builtin_ntp_servers=127.28.112.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 07:57:20.313452 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.305139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:53611 (local address 127.28.112.62:37097)
0809 07:57:20.305270 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:20.305274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:20.306294 (+ 1020us) server_negotiation.cc:407] Connection header received
0809 07:57:20.307665 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:20.307668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:20.307717 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:20.307789 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:20.309008 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.309888 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.311546 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.311828 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.311925 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:20.312995 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:20.312999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:20.313002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:20.313016 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:20.313036 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:20.313044 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:20.313101 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:20.313249 (+ 148us) server_negotiation.cc:299] Negotiation successful
0809 07:57:20.313312 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 07:57:20.314491 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.305489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37097 (local address 127.28.112.1:53611)
0809 07:57:20.306160 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:20.306200 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:20.307458 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:20.307995 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:20.308006 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:20.308110 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:20.308788 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:20.308808 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.310066 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.310070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:20.311435 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:20.311445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.311963 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.311967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:20.312137 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:20.312748 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:57:20.312749 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:20.312788 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:57:20.312850 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:20.313112 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:20.313121 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:20.313127 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 07:57:20.313241 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 07:57:20.313515 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":101,"threads_started":1}
I20260809 07:57:20.315671 25540 heartbeater.cc:347] Connected to a master server at 127.28.112.62:37097
I20260809 07:57:20.315889 25540 heartbeater.cc:464] Registering TS with master...
I20260809 07:57:20.316370 25540 heartbeater.cc:511] Master 127.28.112.62:37097 requested a full tablet report, sending...
I20260809 07:57:20.318068 25250 ts_manager.cc:194] Registered new tserver with Master: b6c6a4930d4845eab1ef6478700a5281 (127.28.112.1:41375)
I20260809 07:57:20.319664 25250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.112.1:53611
W20260809 07:57:20.421840 25548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:20.422116 25548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:20.422152 25548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:20.426029 25548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:20.426075 25548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:20.426133 25548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:57:20.430783 25548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:37097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:20.431775 25548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:20.433101 25548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:20.442130 25555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:20.442126 25556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:20.442133 25559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:20.442219 25548 server_base.cc:1034] running on GCE node
I20260809 07:57:20.442735 25548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:20.443353 25548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:20.444517 25548 hybrid_clock.cc:648] HybridClock initialized: now 1786262240444489 us; error 38 us; skew 500 ppm
I20260809 07:57:20.446221 25548 webserver.cc:460] Webserver started at http://127.28.112.2:41635/ using document root <none> and password file <none>
I20260809 07:57:20.446762 25548 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:20.446827 25548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:20.447016 25548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:20.448994 25548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "1e9c148668ea413b9c2b927af8eff76d"
format_stamp: "Formatted at 2026-08-09 07:57:20 on dist-test-slave-vpvm"
server_key: "3fbce91d38f4cc4c69806741d3fd5b2f"
server_key_iv: "a2071a80135814d8203de8cc2349bef6"
server_key_version: "encryptionkey@0"
I20260809 07:57:20.449513 25548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "1e9c148668ea413b9c2b927af8eff76d"
format_stamp: "Formatted at 2026-08-09 07:57:20 on dist-test-slave-vpvm"
server_key: "3fbce91d38f4cc4c69806741d3fd5b2f"
server_key_iv: "a2071a80135814d8203de8cc2349bef6"
server_key_version: "encryptionkey@0"
I20260809 07:57:20.453009 25548 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:57:20.455294 25567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:20.456374 25548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:57:20.456485 25548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "1e9c148668ea413b9c2b927af8eff76d"
format_stamp: "Formatted at 2026-08-09 07:57:20 on dist-test-slave-vpvm"
server_key: "3fbce91d38f4cc4c69806741d3fd5b2f"
server_key_iv: "a2071a80135814d8203de8cc2349bef6"
server_key_version: "encryptionkey@0"
I20260809 07:57:20.456573 25548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:20.467259 25548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:20.467823 25548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:20.467975 25548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:20.468482 25548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:57:20.469348 25548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:57:20.469396 25548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:20.469445 25548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:57:20.469471 25548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:20.477612 25548 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:33843
I20260809 07:57:20.477666 25732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:33843 every 8 connection(s)
I20260809 07:57:20.478484 25548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:57:20.479038 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 25548
I20260809 07:57:20.479174 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:57:20.479476 29120 external_mini_cluster.cc:1428] Setting key 1596c33712dee66643aa4d6bf9d77105
I20260809 07:57:20.481606 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:37097
--builtin_ntp_servers=127.28.112.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 07:57:20.488195 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.480313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:51541 (local address 127.28.112.62:37097)
0809 07:57:20.480461 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:20.480465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:20.481671 (+ 1206us) server_negotiation.cc:407] Connection header received
0809 07:57:20.483415 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:20.483419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:20.483477 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:20.483552 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:20.484355 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.485130 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.486300 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.486645 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.486763 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:20.487724 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:20.487729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:20.487732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:20.487748 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:20.487772 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:20.487783 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:20.487861 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:20.487970 (+ 109us) server_negotiation.cc:299] Negotiation successful
0809 07:57:20.488040 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 07:57:20.488971 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.480715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37097 (local address 127.28.112.2:51541)
0809 07:57:20.481505 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:20.481555 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:20.483161 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:20.483691 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:20.483698 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:20.483778 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:20.484224 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:20.484236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.485245 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.485247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:20.486170 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:20.486183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.486781 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.486784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:20.486927 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:20.487465 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:57:20.487467 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:20.487505 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:57:20.487567 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:20.487857 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:20.487862 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:20.487865 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 07:57:20.487948 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 07:57:20.488149 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":137,"threads_started":1}
I20260809 07:57:20.490028 25735 heartbeater.cc:347] Connected to a master server at 127.28.112.62:37097
I20260809 07:57:20.490257 25735 heartbeater.cc:464] Registering TS with master...
I20260809 07:57:20.490727 25735 heartbeater.cc:511] Master 127.28.112.62:37097 requested a full tablet report, sending...
I20260809 07:57:20.491766 25250 ts_manager.cc:194] Registered new tserver with Master: 1e9c148668ea413b9c2b927af8eff76d (127.28.112.2:33843)
I20260809 07:57:20.492522 25250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.112.2:51541
W20260809 07:57:20.593243 25738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:20.593515 25738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:20.593554 25738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:20.597414 25738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:20.597487 25738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:20.597537 25738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:57:20.602061 25738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:37097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:20.603001 25738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:20.604308 25738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:20.613467 25744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:20.613582 25738 server_base.cc:1034] running on GCE node
W20260809 07:57:20.613492 25745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:20.613500 25747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:20.614136 25738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:20.614686 25738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:20.615839 25738 hybrid_clock.cc:648] HybridClock initialized: now 1786262240615821 us; error 28 us; skew 500 ppm
I20260809 07:57:20.617514 25738 webserver.cc:460] Webserver started at http://127.28.112.3:35383/ using document root <none> and password file <none>
I20260809 07:57:20.618105 25738 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:20.618168 25738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:20.618356 25738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:20.620388 25738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "9977bee4f4c640c3a0ba2c8642308670"
format_stamp: "Formatted at 2026-08-09 07:57:20 on dist-test-slave-vpvm"
server_key: "a7c6e81106f6546f897feb508ae478ad"
server_key_iv: "44c3e919fd4a0524329547f6fd82cf84"
server_key_version: "encryptionkey@0"
I20260809 07:57:20.620910 25738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "9977bee4f4c640c3a0ba2c8642308670"
format_stamp: "Formatted at 2026-08-09 07:57:20 on dist-test-slave-vpvm"
server_key: "a7c6e81106f6546f897feb508ae478ad"
server_key_iv: "44c3e919fd4a0524329547f6fd82cf84"
server_key_version: "encryptionkey@0"
I20260809 07:57:20.624603 25738 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 07:57:20.626906 25761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:20.628053 25738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:57:20.628151 25738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "9977bee4f4c640c3a0ba2c8642308670"
format_stamp: "Formatted at 2026-08-09 07:57:20 on dist-test-slave-vpvm"
server_key: "a7c6e81106f6546f897feb508ae478ad"
server_key_iv: "44c3e919fd4a0524329547f6fd82cf84"
server_key_version: "encryptionkey@0"
I20260809 07:57:20.628249 25738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:20.639950 25738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:20.640520 25738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:20.640672 25738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:20.641196 25738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:57:20.642102 25738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:57:20.642150 25738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:20.642200 25738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:57:20.642230 25738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:20.650372 25738 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:38739
I20260809 07:57:20.650399 25941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:38739 every 8 connection(s)
I20260809 07:57:20.651186 25738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:57:20.658365 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 25738
I20260809 07:57:20.658468 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:57:20.658728 29120 external_mini_cluster.cc:1428] Setting key 8decc23b2cdc7e45a355c17aa0ce5287
I20260809 07:57:20.659907 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.652850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:58563 (local address 127.28.112.62:37097)
0809 07:57:20.652978 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:20.652982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:20.653978 (+ 996us) server_negotiation.cc:407] Connection header received
0809 07:57:20.655347 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:20.655351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:20.655411 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:20.655482 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:20.656361 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.656888 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.658008 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.658240 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.658333 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:20.659458 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:20.659461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:20.659464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:20.659476 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:20.659495 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:20.659502 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:20.659560 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:20.659706 (+ 146us) server_negotiation.cc:299] Negotiation successful
0809 07:57:20.659768 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 07:57:20.660918 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.653196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:37097 (local address 127.28.112.3:58563)
0809 07:57:20.653813 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:20.653851 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:20.655125 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:20.655659 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:20.655666 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:20.655748 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:20.656210 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:20.656222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.657005 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.657008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:20.657863 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:20.657872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.658383 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.658387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:20.658562 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:20.659195 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:57:20.659197 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:20.659250 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:57:20.659314 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:20.659571 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:20.659580 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:20.659585 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 07:57:20.659699 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 07:57:20.659991 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":121,"threads_started":1}
I20260809 07:57:20.661824 25942 heartbeater.cc:347] Connected to a master server at 127.28.112.62:37097
I20260809 07:57:20.662063 25942 heartbeater.cc:464] Registering TS with master...
I20260809 07:57:20.662550 25942 heartbeater.cc:511] Master 127.28.112.62:37097 requested a full tablet report, sending...
I20260809 07:57:20.663475 25250 ts_manager.cc:194] Registered new tserver with Master: 9977bee4f4c640c3a0ba2c8642308670 (127.28.112.3:38739)
I20260809 07:57:20.663981 25250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.112.3:58563
I20260809 07:57:20.672188 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 07:57:20.677350 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.674050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37554 (local address 127.28.112.62:37097)
0809 07:57:20.674182 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:20.674186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:20.674367 (+ 181us) server_negotiation.cc:407] Connection header received
0809 07:57:20.674454 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:20.674456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:20.674494 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:20.674578 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:20.674837 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.675316 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.676039 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.676236 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.676298 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:20.676979 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:20.676982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:20.676984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:20.676993 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:20.677006 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:20.677013 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:20.677092 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:20.677191 (+ 99us) server_negotiation.cc:299] Negotiation successful
0809 07:57:20.677224 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 07:57:20.679836 25250 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:57:20.682898 25250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:57:20.696633 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.690959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:41375 (local address 127.0.0.1:40726)
0809 07:57:20.691614 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:20.691665 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:20.691801 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:20.692553 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:20.692558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:20.692578 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:20.692683 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:20.692687 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.694007 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.694011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:20.695237 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:20.695248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.696079 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.696083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:20.696200 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:20.696243 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 07:57:20.696355 (+ 112us) client_negotiation.cc:240] Negotiation successful
0809 07:57:20.696469 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260809 07:57:20.696652 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.691111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40726 (local address 127.28.112.1:41375)
0809 07:57:20.691426 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:20.691438 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:20.691948 (+ 510us) server_negotiation.cc:407] Connection header received
0809 07:57:20.692033 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:20.692038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:20.692242 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:20.692390 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:20.692794 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.693866 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.695358 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.695964 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.696071 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:20.696272 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:20.696385 (+ 113us) server_negotiation.cc:299] Negotiation successful
0809 07:57:20.696485 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":109,"threads_started":1}
I20260809 07:57:20.697009 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.691053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:33843 (local address 127.0.0.1:52824)
0809 07:57:20.692265 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:20.692278 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:20.692365 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:20.693053 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:20.693056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:20.693075 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:20.693166 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:20.693172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.694388 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.694391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:20.695719 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:20.695727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.696588 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.696592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:20.696674 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:20.696698 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 07:57:20.696829 (+ 131us) client_negotiation.cc:240] Negotiation successful
0809 07:57:20.696890 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"mutex_wait_us":21,"thread_start_us":92,"threads_started":1}
I20260809 07:57:20.697345 25965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.692093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.28.112.3:38739)
0809 07:57:20.692548 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:20.692558 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:20.692858 (+ 300us) server_negotiation.cc:407] Connection header received
0809 07:57:20.692932 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:20.692936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:20.693101 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:20.693243 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:20.693459 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.694343 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.695779 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.696509 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.696609 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:20.696793 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:20.697078 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 07:57:20.697188 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":169,"threads_started":1}
I20260809 07:57:20.697360 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.691688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:38739 (local address 127.0.0.1:44374)
0809 07:57:20.692682 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:20.692695 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:20.692774 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:20.693243 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:20.693245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:20.693263 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:20.693359 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:20.693366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.694441 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.694444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:20.695647 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:20.695657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.696859 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.696862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:20.696934 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:20.696953 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 07:57:20.697020 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 07:57:20.697241 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"mutex_wait_us":17,"thread_start_us":92,"threads_started":1}
I20260809 07:57:20.697641 25964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.691764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52824 (local address 127.28.112.2:33843)
0809 07:57:20.692604 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:20.692614 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:20.692650 (+ 36us) server_negotiation.cc:407] Connection header received
0809 07:57:20.692729 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:20.692735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:20.692921 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:20.693059 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:20.693292 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.694286 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.695803 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.696464 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.697186 (+ 722us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:20.697366 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:20.697413 (+ 47us) server_negotiation.cc:299] Negotiation successful
0809 07:57:20.697496 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":114,"threads_started":1}
I20260809 07:57:20.699957 25436 tablet_service.cc:1467] Processing CreateTablet for tablet ed2c651e2e16498391fcc74fe1eecc83 (DEFAULT_TABLE table=test-table [id=91f5398ed27b4800a28228bec85e1f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:57:20.699954 25631 tablet_service.cc:1467] Processing CreateTablet for tablet ed2c651e2e16498391fcc74fe1eecc83 (DEFAULT_TABLE table=test-table [id=91f5398ed27b4800a28228bec85e1f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:57:20.700851 25631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed2c651e2e16498391fcc74fe1eecc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:20.700851 25436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed2c651e2e16498391fcc74fe1eecc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:20.704259 25843 tablet_service.cc:1467] Processing CreateTablet for tablet ed2c651e2e16498391fcc74fe1eecc83 (DEFAULT_TABLE table=test-table [id=91f5398ed27b4800a28228bec85e1f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:57:20.705303 25843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed2c651e2e16498391fcc74fe1eecc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:20.707648 25966 tablet_bootstrap.cc:492] T ed2c651e2e16498391fcc74fe1eecc83 P b6c6a4930d4845eab1ef6478700a5281: Bootstrap starting.
I20260809 07:57:20.708359 25969 tablet_bootstrap.cc:492] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d: Bootstrap starting.
I20260809 07:57:20.709821 25966 tablet_bootstrap.cc:654] T ed2c651e2e16498391fcc74fe1eecc83 P b6c6a4930d4845eab1ef6478700a5281: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:20.710435 25969 tablet_bootstrap.cc:654] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:20.710572 25966 log.cc:824] T ed2c651e2e16498391fcc74fe1eecc83 P b6c6a4930d4845eab1ef6478700a5281: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:20.711254 25969 log.cc:824] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:20.712476 25966 tablet_bootstrap.cc:492] T ed2c651e2e16498391fcc74fe1eecc83 P b6c6a4930d4845eab1ef6478700a5281: No bootstrap required, opened a new log
I20260809 07:57:20.712705 25966 ts_tablet_manager.cc:1397] T ed2c651e2e16498391fcc74fe1eecc83 P b6c6a4930d4845eab1ef6478700a5281: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 07:57:20.712973 25976 tablet_bootstrap.cc:492] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670: Bootstrap starting.
I20260809 07:57:20.713778 25969 tablet_bootstrap.cc:492] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d: No bootstrap required, opened a new log
I20260809 07:57:20.713995 25969 ts_tablet_manager.cc:1397] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 07:57:20.715082 25976 tablet_bootstrap.cc:654] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:20.715853 25976 log.cc:824] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:20.716094 25966 raft_consensus.cc:348] T ed2c651e2e16498391fcc74fe1eecc83 P b6c6a4930d4845eab1ef6478700a5281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c6a4930d4845eab1ef6478700a5281" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41375 } } peers { permanent_uuid: "1e9c148668ea413b9c2b927af8eff76d" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 33843 } } peers { permanent_uuid: "9977bee4f4c640c3a0ba2c8642308670" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38739 } }
I20260809 07:57:20.716351 25966 raft_consensus.cc:374] T ed2c651e2e16498391fcc74fe1eecc83 P b6c6a4930d4845eab1ef6478700a5281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:20.716440 25966 raft_consensus.cc:729] T ed2c651e2e16498391fcc74fe1eecc83 P b6c6a4930d4845eab1ef6478700a5281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c6a4930d4845eab1ef6478700a5281, State: Initialized, Role: FOLLOWER
I20260809 07:57:20.716873 25966 consensus_queue.cc:260] T ed2c651e2e16498391fcc74fe1eecc83 P b6c6a4930d4845eab1ef6478700a5281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c6a4930d4845eab1ef6478700a5281" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41375 } } peers { permanent_uuid: "1e9c148668ea413b9c2b927af8eff76d" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 33843 } } peers { permanent_uuid: "9977bee4f4c640c3a0ba2c8642308670" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38739 } }
I20260809 07:57:20.717460 25540 heartbeater.cc:503] Master 127.28.112.62:37097 was elected leader, sending a full tablet report...
I20260809 07:57:20.717522 25969 raft_consensus.cc:348] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c6a4930d4845eab1ef6478700a5281" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41375 } } peers { permanent_uuid: "1e9c148668ea413b9c2b927af8eff76d" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 33843 } } peers { permanent_uuid: "9977bee4f4c640c3a0ba2c8642308670" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38739 } }
I20260809 07:57:20.717744 25976 tablet_bootstrap.cc:492] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670: No bootstrap required, opened a new log
I20260809 07:57:20.717752 25969 raft_consensus.cc:374] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:20.717797 25969 raft_consensus.cc:729] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e9c148668ea413b9c2b927af8eff76d, State: Initialized, Role: FOLLOWER
I20260809 07:57:20.717813 25966 ts_tablet_manager.cc:1428] T ed2c651e2e16498391fcc74fe1eecc83 P b6c6a4930d4845eab1ef6478700a5281: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260809 07:57:20.717912 25976 ts_tablet_manager.cc:1397] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260809 07:57:20.718259 25969 consensus_queue.cc:260] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c6a4930d4845eab1ef6478700a5281" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41375 } } peers { permanent_uuid: "1e9c148668ea413b9c2b927af8eff76d" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 33843 } } peers { permanent_uuid: "9977bee4f4c640c3a0ba2c8642308670" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38739 } }
I20260809 07:57:20.718907 25735 heartbeater.cc:503] Master 127.28.112.62:37097 was elected leader, sending a full tablet report...
I20260809 07:57:20.719223 25969 ts_tablet_manager.cc:1428] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 07:57:20.722911 25976 raft_consensus.cc:348] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c6a4930d4845eab1ef6478700a5281" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41375 } } peers { permanent_uuid: "1e9c148668ea413b9c2b927af8eff76d" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 33843 } } peers { permanent_uuid: "9977bee4f4c640c3a0ba2c8642308670" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38739 } }
I20260809 07:57:20.723093 25976 raft_consensus.cc:374] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:20.723165 25976 raft_consensus.cc:729] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9977bee4f4c640c3a0ba2c8642308670, State: Initialized, Role: FOLLOWER
I20260809 07:57:20.723500 25976 consensus_queue.cc:260] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c6a4930d4845eab1ef6478700a5281" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41375 } } peers { permanent_uuid: "1e9c148668ea413b9c2b927af8eff76d" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 33843 } } peers { permanent_uuid: "9977bee4f4c640c3a0ba2c8642308670" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38739 } }
I20260809 07:57:20.724050 25942 heartbeater.cc:503] Master 127.28.112.62:37097 was elected leader, sending a full tablet report...
I20260809 07:57:20.724294 25976 ts_tablet_manager.cc:1428] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
W20260809 07:57:20.729441 25736 tablet.cc:2369] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:57:20.740880 25981 raft_consensus.cc:482] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:57:20.741047 25981 raft_consensus.cc:504] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c6a4930d4845eab1ef6478700a5281" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41375 } } peers { permanent_uuid: "1e9c148668ea413b9c2b927af8eff76d" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 33843 } } peers { permanent_uuid: "9977bee4f4c640c3a0ba2c8642308670" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38739 } }
I20260809 07:57:20.742111 25981 leader_election.cc:288] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6c6a4930d4845eab1ef6478700a5281 (127.28.112.1:41375), 1e9c148668ea413b9c2b927af8eff76d (127.28.112.2:33843)
I20260809 07:57:20.746677 25964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.742405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:38717 (local address 127.28.112.2:33843)
0809 07:57:20.742516 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:20.742520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:20.742735 (+ 215us) server_negotiation.cc:407] Connection header received
0809 07:57:20.742804 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:20.742807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:20.742851 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:20.742919 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:20.743169 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.743883 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.745483 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.746104 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.746213 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:20.746262 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:20.746489 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 07:57:20.746555 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 07:57:20.746677 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.742389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:58527 (local address 127.28.112.1:41375)
0809 07:57:20.742489 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:20.742493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:20.742510 (+ 17us) server_negotiation.cc:407] Connection header received
0809 07:57:20.742685 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:20.742688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:20.742742 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:20.742815 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:20.743069 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.743865 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.745483 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.746096 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.746213 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:20.746262 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:20.746497 (+ 235us) server_negotiation.cc:299] Negotiation successful
0809 07:57:20.746560 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 07:57:20.746701 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.742284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:41375 (local address 127.28.112.3:58527)
0809 07:57:20.742458 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:20.742472 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:20.742556 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:20.742838 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:20.742840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:20.742853 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:20.742965 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:20.742973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.743970 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.743973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:20.745368 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:20.745380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.746211 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.746215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:20.746369 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:20.746402 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 07:57:20.746469 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 07:57:20.746549 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":147}
I20260809 07:57:20.746716 25982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.742326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:33843 (local address 127.28.112.3:38717)
0809 07:57:20.742650 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:20.742663 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:20.742738 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:20.742938 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:20.742940 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:20.742953 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:20.743057 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:20.743063 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.743970 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.743973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:20.745368 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:20.745380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.746216 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.746219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:20.746353 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:20.746397 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 07:57:20.746473 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 07:57:20.746549 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":131,"thread_start_us":104,"threads_started":1}
I20260809 07:57:20.747300 25458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2c651e2e16498391fcc74fe1eecc83" candidate_uuid: "9977bee4f4c640c3a0ba2c8642308670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6c6a4930d4845eab1ef6478700a5281" is_pre_election: true
I20260809 07:57:20.747385 25656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2c651e2e16498391fcc74fe1eecc83" candidate_uuid: "9977bee4f4c640c3a0ba2c8642308670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9c148668ea413b9c2b927af8eff76d" is_pre_election: true
I20260809 07:57:20.747568 25458 raft_consensus.cc:2454] T ed2c651e2e16498391fcc74fe1eecc83 P b6c6a4930d4845eab1ef6478700a5281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9977bee4f4c640c3a0ba2c8642308670 in term 0.
I20260809 07:57:20.747638 25656 raft_consensus.cc:2454] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9977bee4f4c640c3a0ba2c8642308670 in term 0.
I20260809 07:57:20.748013 25764 leader_election.cc:302] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1e9c148668ea413b9c2b927af8eff76d, 9977bee4f4c640c3a0ba2c8642308670, b6c6a4930d4845eab1ef6478700a5281; no voters:
I20260809 07:57:20.748377 25981 raft_consensus.cc:2781] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:57:20.748453 25981 raft_consensus.cc:482] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:57:20.748482 25981 raft_consensus.cc:3037] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:20.749404 25981 raft_consensus.cc:504] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c6a4930d4845eab1ef6478700a5281" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41375 } } peers { permanent_uuid: "1e9c148668ea413b9c2b927af8eff76d" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 33843 } } peers { permanent_uuid: "9977bee4f4c640c3a0ba2c8642308670" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38739 } }
I20260809 07:57:20.749697 25981 leader_election.cc:288] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [CANDIDATE]: Term 1 election: Requested vote from peers b6c6a4930d4845eab1ef6478700a5281 (127.28.112.1:41375), 1e9c148668ea413b9c2b927af8eff76d (127.28.112.2:33843)
I20260809 07:57:20.750041 25656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2c651e2e16498391fcc74fe1eecc83" candidate_uuid: "9977bee4f4c640c3a0ba2c8642308670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9c148668ea413b9c2b927af8eff76d"
I20260809 07:57:20.750051 25458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2c651e2e16498391fcc74fe1eecc83" candidate_uuid: "9977bee4f4c640c3a0ba2c8642308670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6c6a4930d4845eab1ef6478700a5281"
I20260809 07:57:20.750166 25656 raft_consensus.cc:3037] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:20.750173 25458 raft_consensus.cc:3037] T ed2c651e2e16498391fcc74fe1eecc83 P b6c6a4930d4845eab1ef6478700a5281 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:20.751314 25458 raft_consensus.cc:2454] T ed2c651e2e16498391fcc74fe1eecc83 P b6c6a4930d4845eab1ef6478700a5281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9977bee4f4c640c3a0ba2c8642308670 in term 1.
I20260809 07:57:20.751338 25656 raft_consensus.cc:2454] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9977bee4f4c640c3a0ba2c8642308670 in term 1.
I20260809 07:57:20.751586 25764 leader_election.cc:302] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9977bee4f4c640c3a0ba2c8642308670, b6c6a4930d4845eab1ef6478700a5281; no voters:
I20260809 07:57:20.751785 25981 raft_consensus.cc:2781] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:20.751958 25981 raft_consensus.cc:686] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [term 1 LEADER]: Becoming Leader. State: Replica: 9977bee4f4c640c3a0ba2c8642308670, State: Running, Role: LEADER
I20260809 07:57:20.752270 25981 consensus_queue.cc:237] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c6a4930d4845eab1ef6478700a5281" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41375 } } peers { permanent_uuid: "1e9c148668ea413b9c2b927af8eff76d" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 33843 } } peers { permanent_uuid: "9977bee4f4c640c3a0ba2c8642308670" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38739 } }
I20260809 07:57:20.755438 25250 catalog_manager.cc:5495] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9977bee4f4c640c3a0ba2c8642308670 (127.28.112.3). New cstate: current_term: 1 leader_uuid: "9977bee4f4c640c3a0ba2c8642308670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c6a4930d4845eab1ef6478700a5281" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e9c148668ea413b9c2b927af8eff76d" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 33843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9977bee4f4c640c3a0ba2c8642308670" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 38739 } health_report { overall_health: HEALTHY } } }
I20260809 07:57:20.774041 25965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.770691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44376 (local address 127.28.112.3:38739)
0809 07:57:20.770798 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:20.770803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:20.770815 (+ 12us) server_negotiation.cc:407] Connection header received
0809 07:57:20.770929 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:20.770932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:20.770983 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:20.771066 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:57:20.771276 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.771757 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.772605 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:20.772820 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:20.772892 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:20.773235 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:57:20.773632 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:57:20.773772 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:20.773863 (+ 91us) server_negotiation.cc:299] Negotiation successful
0809 07:57:20.773893 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 07:57:20.779378 25458 raft_consensus.cc:1261] T ed2c651e2e16498391fcc74fe1eecc83 P b6c6a4930d4845eab1ef6478700a5281 [term 1 FOLLOWER]: Refusing update from remote peer 9977bee4f4c640c3a0ba2c8642308670: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:57:20.779378 25656 raft_consensus.cc:1261] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d [term 1 FOLLOWER]: Refusing update from remote peer 9977bee4f4c640c3a0ba2c8642308670: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:57:20.780023 25981 consensus_queue.cc:1035] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6c6a4930d4845eab1ef6478700a5281" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 41375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:57:20.780206 25983 consensus_queue.cc:1035] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e9c148668ea413b9c2b927af8eff76d" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:57:20.787756 25989 mvcc.cc:205] Tried to move back new op lower bound from 7316530138226040832 to 7316530138123816960. Current Snapshot: MvccSnapshot[applied={T|T < 7316530138226040832}]
I20260809 07:57:20.787900 25988 mvcc.cc:205] Tried to move back new op lower bound from 7316530138226040832 to 7316530138123816960. Current Snapshot: MvccSnapshot[applied={T|T < 7316530138226040832}]
I20260809 07:57:20.796900 25249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:57:20.797041 25249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:57:20.799257 25249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33a487e3cc8140b19d804c1106d96cdb: Sending DeleteTablet for 3 replicas of tablet ed2c651e2e16498391fcc74fe1eecc83
I20260809 07:57:20.800083 25843 tablet_service.cc:1514] Processing DeleteTablet for tablet ed2c651e2e16498391fcc74fe1eecc83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:57:20 UTC) from {username='slave'} at 127.0.0.1:44374
I20260809 07:57:20.800056 25631 tablet_service.cc:1514] Processing DeleteTablet for tablet ed2c651e2e16498391fcc74fe1eecc83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:57:20 UTC) from {username='slave'} at 127.0.0.1:52824
I20260809 07:57:20.800596 26003 tablet_replica.cc:330] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670: stopping tablet replica
I20260809 07:57:20.800747 25436 tablet_service.cc:1514] Processing DeleteTablet for tablet ed2c651e2e16498391fcc74fe1eecc83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:57:20 UTC) from {username='slave'} at 127.0.0.1:40726
I20260809 07:57:20.800985 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 25338
I20260809 07:57:20.803704 26003 raft_consensus.cc:2229] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:57:20.803781 26004 tablet_replica.cc:330] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d: stopping tablet replica
I20260809 07:57:20.804134 26003 raft_consensus.cc:2258] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:57:20.804137 26004 raft_consensus.cc:2229] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:57:20.804422 26004 raft_consensus.cc:2258] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:57:20.805862 26004 ts_tablet_manager.cc:1905] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:57:20.807495 26003 ts_tablet_manager.cc:1905] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:57:20.808679 26004 ts_tablet_manager.cc:1918] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:57:20.808882 26004 log.cc:1196] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/ed2c651e2e16498391fcc74fe1eecc83
I20260809 07:57:20.809384 26004 ts_tablet_manager.cc:1939] T ed2c651e2e16498391fcc74fe1eecc83 P 1e9c148668ea413b9c2b927af8eff76d: Deleting consensus metadata
I20260809 07:57:20.810746 26003 ts_tablet_manager.cc:1918] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:57:20.810762 25228 catalog_manager.cc:4841] TS 1e9c148668ea413b9c2b927af8eff76d (127.28.112.2:33843): tablet ed2c651e2e16498391fcc74fe1eecc83 (table test-table [id=91f5398ed27b4800a28228bec85e1f3f]) successfully deleted
I20260809 07:57:20.810897 26003 log.cc:1196] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/ed2c651e2e16498391fcc74fe1eecc83
I20260809 07:57:20.811173 26003 ts_tablet_manager.cc:1939] T ed2c651e2e16498391fcc74fe1eecc83 P 9977bee4f4c640c3a0ba2c8642308670: Deleting consensus metadata
I20260809 07:57:20.812359 25230 catalog_manager.cc:4841] TS 9977bee4f4c640c3a0ba2c8642308670 (127.28.112.3:38739): tablet ed2c651e2e16498391fcc74fe1eecc83 (table test-table [id=91f5398ed27b4800a28228bec85e1f3f]) successfully deleted
W20260809 07:57:20.815816 25230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.112.1:41375 (error 108)
I20260809 07:57:20.815891 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 25548
I20260809 07:57:20.822490 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:20.822116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:41375 (local address 127.0.0.1:40728)
0809 07:57:20.822312 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:20.822387 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:41375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260809 07:57:20.822687 25230 catalog_manager.cc:4584] TS b6c6a4930d4845eab1ef6478700a5281 (127.28.112.1:41375): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed2c651e2e16498391fcc74fe1eecc83: Network error: Client connection negotiation failed: client connection to 127.28.112.1:41375: connect: Connection refused (error 111)
I20260809 07:57:20.828565 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 25738
I20260809 07:57:20.840669 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 25208
2026-08-09T07:57:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1012 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:20 dist-test-slave-vpvm krb5kdc[26009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:20 dist-test-slave-vpvm krb5kdc[26009](info): set up 2 sockets
Aug 09 07:57:20 dist-test-slave-vpvm krb5kdc[26009](info): commencing operation
krb5kdc: starting...
W20260809 07:57:21.967779 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:22 dist-test-slave-vpvm krb5kdc[26009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:57:22.086544 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:35083
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:38435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:35083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:22.195868 26030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:22.196120 26030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:22.196185 26030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:22.200084 26030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:22.200142 26030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:22.200170 26030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:22.200191 26030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:22.200222 26030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:22.205353 26030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:35083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:35083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:22.206367 26030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:22.207361 26030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:22.215401 26035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:22.215411 26036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:22.215411 26041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:22.215703 26030 server_base.cc:1034] running on GCE node
I20260809 07:57:22.216221 26030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:22.217255 26030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:22.218410 26030 hybrid_clock.cc:648] HybridClock initialized: now 1786262242218390 us; error 30 us; skew 500 ppm
Aug 09 07:57:22 dist-test-slave-vpvm krb5kdc[26009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262242, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:22.220950 26030 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:57:22.221797 26030 webserver.cc:460] Webserver started at http://127.28.112.62:44643/ using document root <none> and password file <none>
I20260809 07:57:22.222338 26030 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:22.222404 26030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:22.222615 26030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:22.224551 26030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "dc69105fcca94ec484bdf7a4343b474d"
format_stamp: "Formatted at 2026-08-09 07:57:22 on dist-test-slave-vpvm"
server_key: "e5bb1872e1b7c8f4bcd341e004d92af2"
server_key_iv: "6631d14b658ff2ca09308c45f522e66e"
server_key_version: "encryptionkey@0"
I20260809 07:57:22.225080 26030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "dc69105fcca94ec484bdf7a4343b474d"
format_stamp: "Formatted at 2026-08-09 07:57:22 on dist-test-slave-vpvm"
server_key: "e5bb1872e1b7c8f4bcd341e004d92af2"
server_key_iv: "6631d14b658ff2ca09308c45f522e66e"
server_key_version: "encryptionkey@0"
I20260809 07:57:22.228562 26030 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 07:57:22.230765 26052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:22.231912 26030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:57:22.232021 26030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "dc69105fcca94ec484bdf7a4343b474d"
format_stamp: "Formatted at 2026-08-09 07:57:22 on dist-test-slave-vpvm"
server_key: "e5bb1872e1b7c8f4bcd341e004d92af2"
server_key_iv: "6631d14b658ff2ca09308c45f522e66e"
server_key_version: "encryptionkey@0"
I20260809 07:57:22.232111 26030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:22.255192 26030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:22.258118 26030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:22.258286 26030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:22.264912 26030 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:35083
I20260809 07:57:22.264923 26132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:35083 every 8 connection(s)
I20260809 07:57:22.265805 26030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:22.268554 26133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:22.272034 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 26030
I20260809 07:57:22.272181 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:22.272429 29120 external_mini_cluster.cc:1428] Setting key cf913258cb9de2de96f96bca2ef300d8
I20260809 07:57:22.274248 26133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d: Bootstrap starting.
I20260809 07:57:22.276669 26133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:22.277374 26133 log.cc:824] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:22.279400 26133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d: No bootstrap required, opened a new log
Aug 09 07:57:22 dist-test-slave-vpvm krb5kdc[26009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:57:22.283032 26133 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc69105fcca94ec484bdf7a4343b474d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 35083 } }
I20260809 07:57:22.283327 26133 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:22.283397 26133 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc69105fcca94ec484bdf7a4343b474d, State: Initialized, Role: FOLLOWER
I20260809 07:57:22.283885 26133 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc69105fcca94ec484bdf7a4343b474d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 35083 } }
I20260809 07:57:22.284044 26133 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:22.284121 26133 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:22.284227 26133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:22.285249 26133 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc69105fcca94ec484bdf7a4343b474d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 35083 } }
I20260809 07:57:22.285615 26133 leader_election.cc:302] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc69105fcca94ec484bdf7a4343b474d; no voters:
I20260809 07:57:22.285986 26133 leader_election.cc:288] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:22.286072 26141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:22.286326 26141 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d [term 1 LEADER]: Becoming Leader. State: Replica: dc69105fcca94ec484bdf7a4343b474d, State: Running, Role: LEADER
I20260809 07:57:22.286680 26141 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc69105fcca94ec484bdf7a4343b474d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 35083 } }
I20260809 07:57:22.287199 26139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:22.273735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38306 (local address 127.28.112.62:35083)
0809 07:57:22.274154 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:22.274171 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:22.274208 (+ 37us) server_negotiation.cc:407] Connection header received
0809 07:57:22.274830 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:22.274847 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:22.275160 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:22.275418 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:22.276296 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.277290 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.278325 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.278778 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.278891 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:22.281585 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:22.281592 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:22.281609 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:22.281642 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:22.283976 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:22.284500 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:22.284506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:22.284512 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:22.284645 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:22.284867 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:22.284870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:22.284872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:22.285406 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:22.285586 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:22.285927 (+ 341us) server_negotiation.cc:299] Negotiation successful
0809 07:57:22.286272 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":100,"threads_started":1}
I20260809 07:57:22.287297 26133 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:22.287822 26141 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc69105fcca94ec484bdf7a4343b474d. Latest consensus state: current_term: 1 leader_uuid: "dc69105fcca94ec484bdf7a4343b474d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc69105fcca94ec484bdf7a4343b474d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 35083 } } }
I20260809 07:57:22.288053 26141 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:22.288216 26142 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc69105fcca94ec484bdf7a4343b474d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc69105fcca94ec484bdf7a4343b474d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 35083 } } }
I20260809 07:57:22.288311 26142 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d [sys.catalog]: This master's current role is: LEADER
W20260809 07:57:22.291783 26161 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:57:22.291850 26161 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:57:22.291975 26152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:22.292893 26152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:22.298339 26152 catalog_manager.cc:1347] Generated new cluster ID: a1b5152fc737422cb53a9d73824acf9f
I20260809 07:57:22.298405 26152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:22.316941 26152 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:22.318490 26152 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:22.323804 26152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d: Generated new TSK 0
I20260809 07:57:22.324429 26152 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.112.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
Entry for principal kudu/127.28.112.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab.
I20260809 07:57:22.353807 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:35083
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:38435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:22.463474 26169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:22.463708 26169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:22.463754 26169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:22.467666 26169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:22.467715 26169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:22.467782 26169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:57:22.472812 26169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.1.keytab
--principal=kudu/127.28.112.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:35083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:22.473766 26169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:22.474747 26169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 07:57:22.483559 26169 server_base.cc:1034] running on GCE node
W20260809 07:57:22.483474 26179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:22.483498 26176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:22.483482 26177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:22.484122 26169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:22.484696 26169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:22.485844 26169 hybrid_clock.cc:648] HybridClock initialized: now 1786262242485828 us; error 29 us; skew 500 ppm
Aug 09 07:57:22 dist-test-slave-vpvm krb5kdc[26009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262242, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:22.488516 26169 init.cc:377] Logged in from keytab as kudu/127.28.112.1@KRBTEST.COM (short username kudu)
I20260809 07:57:22.489313 26169 webserver.cc:460] Webserver started at http://127.28.112.1:38571/ using document root <none> and password file <none>
I20260809 07:57:22.489838 26169 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:22.489897 26169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:22.490099 26169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:22.491964 26169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "2f95363e8db34360934249d055d84dfd"
format_stamp: "Formatted at 2026-08-09 07:57:22 on dist-test-slave-vpvm"
server_key: "8c59769550c43a56118471912b031eb7"
server_key_iv: "eb6b8bdf65d08f75dc2427a4d33bbd75"
server_key_version: "encryptionkey@0"
I20260809 07:57:22.492456 26169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "2f95363e8db34360934249d055d84dfd"
format_stamp: "Formatted at 2026-08-09 07:57:22 on dist-test-slave-vpvm"
server_key: "8c59769550c43a56118471912b031eb7"
server_key_iv: "eb6b8bdf65d08f75dc2427a4d33bbd75"
server_key_version: "encryptionkey@0"
I20260809 07:57:22.495890 26169 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 07:57:22.498080 26190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:22.499193 26169 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:57:22.499305 26169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "2f95363e8db34360934249d055d84dfd"
format_stamp: "Formatted at 2026-08-09 07:57:22 on dist-test-slave-vpvm"
server_key: "8c59769550c43a56118471912b031eb7"
server_key_iv: "eb6b8bdf65d08f75dc2427a4d33bbd75"
server_key_version: "encryptionkey@0"
I20260809 07:57:22.499394 26169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:22.509788 26169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:22.512676 26169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:22.512836 26169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:22.513360 26169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:57:22.514168 26169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:57:22.514215 26169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:22.514266 26169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:57:22.514292 26169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:22.522285 26169 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:38163
I20260809 07:57:22.522317 26373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:38163 every 8 connection(s)
I20260809 07:57:22.523100 26169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:57:22.529915 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 26169
I20260809 07:57:22.530081 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:57:22.530411 29120 external_mini_cluster.cc:1428] Setting key a6735cbf7aee107c3bae5bbb0129349d
Aug 09 07:57:22 dist-test-slave-vpvm krb5kdc[26009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262242, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.1@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:57:22.536075 26139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:22.524742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:51387 (local address 127.28.112.62:35083)
0809 07:57:22.524859 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:22.524863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:22.525920 (+ 1057us) server_negotiation.cc:407] Connection header received
0809 07:57:22.526946 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:22.526949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:22.527000 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:22.527061 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:22.528130 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.528661 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.529705 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.530065 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.530171 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:22.532613 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:22.532617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:22.532620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:22.532648 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:22.533988 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:22.534534 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:22.534536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:22.534538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:22.534616 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:22.535019 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:22.535021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:22.535022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:22.535189 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:22.535260 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:22.535858 (+ 598us) server_negotiation.cc:299] Negotiation successful
0809 07:57:22.535957 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 07:57:22.537154 26376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:22.525054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:35083 (local address 127.28.112.1:51387)
0809 07:57:22.525759 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:22.525811 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:22.526748 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:22.527216 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:22.527223 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:22.527554 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:22.528019 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:22.528033 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.528751 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.528753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:22.529595 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:22.529603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.530221 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.530225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:22.530392 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:22.530939 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:22.530961 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:22.532440 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:22.534095 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:22.534101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:22.534116 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:22.534446 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:22.534745 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:22.534748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:22.534750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:22.534930 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:22.535286 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:22.535292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:22.535580 (+ 288us) client_negotiation.cc:769] Sending connection context
0809 07:57:22.535822 (+ 242us) client_negotiation.cc:240] Negotiation successful
0809 07:57:22.536140 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":104,"threads_started":1}
I20260809 07:57:22.538323 26374 heartbeater.cc:347] Connected to a master server at 127.28.112.62:35083
I20260809 07:57:22.538590 26374 heartbeater.cc:464] Registering TS with master...
I20260809 07:57:22.539168 26374 heartbeater.cc:511] Master 127.28.112.62:35083 requested a full tablet report, sending...
I20260809 07:57:22.540673 26078 ts_manager.cc:194] Registered new tserver with Master: 2f95363e8db34360934249d055d84dfd (127.28.112.1:38163)
WARNING: no policy specified for kudu/127.28.112.2@KRBTEST.COM; defaulting to no policy
I20260809 07:57:22.541801 26078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.1@KRBTEST.COM'} at 127.28.112.1:51387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
Entry for principal kudu/127.28.112.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab.
I20260809 07:57:22.556906 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:35083
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:38435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:22.666220 26379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:22.666461 26379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:22.666507 26379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:22.670435 26379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:22.670485 26379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:22.670555 26379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:57:22.675659 26379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.2.keytab
--principal=kudu/127.28.112.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:35083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:22.676648 26379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:22.677690 26379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:22.687250 26394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:22.687270 26390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:22.687320 26379 server_base.cc:1034] running on GCE node
W20260809 07:57:22.687234 26388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:22.687899 26379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:22.688505 26379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:22.689656 26379 hybrid_clock.cc:648] HybridClock initialized: now 1786262242689636 us; error 34 us; skew 500 ppm
Aug 09 07:57:22 dist-test-slave-vpvm krb5kdc[26009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262242, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:22.692332 26379 init.cc:377] Logged in from keytab as kudu/127.28.112.2@KRBTEST.COM (short username kudu)
I20260809 07:57:22.693128 26379 webserver.cc:460] Webserver started at http://127.28.112.2:43087/ using document root <none> and password file <none>
I20260809 07:57:22.693645 26379 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:22.693710 26379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:22.693887 26379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:22.695808 26379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "dc56db8252cd4aff8a6cb94693087673"
format_stamp: "Formatted at 2026-08-09 07:57:22 on dist-test-slave-vpvm"
server_key: "f4403da7623e8cb28ed81dfd5613f924"
server_key_iv: "1f3d2d1949fe4143cbf30b4daf913534"
server_key_version: "encryptionkey@0"
I20260809 07:57:22.696321 26379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "dc56db8252cd4aff8a6cb94693087673"
format_stamp: "Formatted at 2026-08-09 07:57:22 on dist-test-slave-vpvm"
server_key: "f4403da7623e8cb28ed81dfd5613f924"
server_key_iv: "1f3d2d1949fe4143cbf30b4daf913534"
server_key_version: "encryptionkey@0"
I20260809 07:57:22.699786 26379 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 07:57:22.702076 26409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:22.703275 26379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:57:22.703378 26379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "dc56db8252cd4aff8a6cb94693087673"
format_stamp: "Formatted at 2026-08-09 07:57:22 on dist-test-slave-vpvm"
server_key: "f4403da7623e8cb28ed81dfd5613f924"
server_key_iv: "1f3d2d1949fe4143cbf30b4daf913534"
server_key_version: "encryptionkey@0"
I20260809 07:57:22.703467 26379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:22.717159 26379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:22.719769 26379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:22.719937 26379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:22.720481 26379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:57:22.721267 26379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:57:22.721314 26379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:22.721367 26379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:57:22.721395 26379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:22.729579 26379 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:46083
I20260809 07:57:22.729627 26578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:46083 every 8 connection(s)
I20260809 07:57:22.730439 26379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:57:22.732543 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 26379
I20260809 07:57:22.732673 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:57:22.732956 29120 external_mini_cluster.cc:1428] Setting key de6a178d4814a698a4f237d77c39d30e
Aug 09 07:57:22 dist-test-slave-vpvm krb5kdc[26009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262242, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.2@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.112.3@KRBTEST.COM; defaulting to no policy
I20260809 07:57:22.743803 26139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:22.732148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:52951 (local address 127.28.112.62:35083)
0809 07:57:22.732271 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:22.732274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:22.733349 (+ 1075us) server_negotiation.cc:407] Connection header received
0809 07:57:22.734230 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:22.734233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:22.734286 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:22.734357 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:22.735834 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.736410 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.737514 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.737742 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.737830 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:22.740291 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:22.740295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:22.740298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:22.740329 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:22.741780 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:22.742279 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:22.742284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:22.742285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:22.742381 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:22.742692 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:22.742696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:22.742698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:22.742934 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:22.743036 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:22.743495 (+ 459us) server_negotiation.cc:299] Negotiation successful
0809 07:57:22.743620 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 07:57:22.744441 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:22.732489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:35083 (local address 127.28.112.2:52951)
0809 07:57:22.733194 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:22.733246 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:22.734051 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:22.734570 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:22.734579 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:22.735029 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:22.735702 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:22.735721 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.736531 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.736535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:22.737393 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:22.737403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.737883 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.737887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:22.738095 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:22.738619 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:22.738634 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:22.740124 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:22.741890 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:22.741895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:22.741906 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:22.742186 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:22.742466 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:22.742468 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:22.742469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:22.742595 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:22.743025 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:22.743030 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:22.743256 (+ 226us) client_negotiation.cc:769] Sending connection context
0809 07:57:22.743396 (+ 140us) client_negotiation.cc:240] Negotiation successful
0809 07:57:22.743638 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":126,"threads_started":1}
I20260809 07:57:22.745524 26582 heartbeater.cc:347] Connected to a master server at 127.28.112.62:35083
I20260809 07:57:22.745750 26582 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.3@KRBTEST.COM" created.
I20260809 07:57:22.746261 26582 heartbeater.cc:511] Master 127.28.112.62:35083 requested a full tablet report, sending...
I20260809 07:57:22.747272 26078 ts_manager.cc:194] Registered new tserver with Master: dc56db8252cd4aff8a6cb94693087673 (127.28.112.2:46083)
I20260809 07:57:22.747905 26078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.2@KRBTEST.COM'} at 127.28.112.2:52951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
Entry for principal kudu/127.28.112.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab.
I20260809 07:57:22.760442 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:35083
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:38435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:22.868741 26589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:22.868996 26589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:22.869041 26589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:22.872982 26589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:22.873035 26589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:22.873104 26589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:57:22.878188 26589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.3.keytab
--principal=kudu/127.28.112.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:35083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:22.879174 26589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:22.880159 26589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:22.889636 26595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:22.889606 26596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:22.889606 26598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:22.889655 26589 server_base.cc:1034] running on GCE node
I20260809 07:57:22.890182 26589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:22.890782 26589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:22.891937 26589 hybrid_clock.cc:648] HybridClock initialized: now 1786262242891921 us; error 32 us; skew 500 ppm
Aug 09 07:57:22 dist-test-slave-vpvm krb5kdc[26009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262242, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:22.894467 26589 init.cc:377] Logged in from keytab as kudu/127.28.112.3@KRBTEST.COM (short username kudu)
I20260809 07:57:22.895252 26589 webserver.cc:460] Webserver started at http://127.28.112.3:35711/ using document root <none> and password file <none>
I20260809 07:57:22.895785 26589 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:22.895852 26589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:22.896049 26589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:22.898032 26589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "aa156f6ee78f4249b3bad7f6dc10e6b4"
format_stamp: "Formatted at 2026-08-09 07:57:22 on dist-test-slave-vpvm"
server_key: "f154233499f1ef6f9ab32895a751939f"
server_key_iv: "550831d2ad8d2a6ce94e5f77991b7278"
server_key_version: "encryptionkey@0"
I20260809 07:57:22.898564 26589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "aa156f6ee78f4249b3bad7f6dc10e6b4"
format_stamp: "Formatted at 2026-08-09 07:57:22 on dist-test-slave-vpvm"
server_key: "f154233499f1ef6f9ab32895a751939f"
server_key_iv: "550831d2ad8d2a6ce94e5f77991b7278"
server_key_version: "encryptionkey@0"
I20260809 07:57:22.901985 26589 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 07:57:22.904306 26609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:22.905417 26589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:57:22.905525 26589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "aa156f6ee78f4249b3bad7f6dc10e6b4"
format_stamp: "Formatted at 2026-08-09 07:57:22 on dist-test-slave-vpvm"
server_key: "f154233499f1ef6f9ab32895a751939f"
server_key_iv: "550831d2ad8d2a6ce94e5f77991b7278"
server_key_version: "encryptionkey@0"
I20260809 07:57:22.905609 26589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:22.920852 26589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:22.923820 26589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:22.923990 26589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:22.924530 26589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:57:22.925312 26589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:57:22.925369 26589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:22.925422 26589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:57:22.925448 26589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:22.933565 26589 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:36033
I20260809 07:57:22.933595 26783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:36033 every 8 connection(s)
I20260809 07:57:22.934429 26589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:57:22.935947 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 26589
I20260809 07:57:22.936071 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:57:22.936389 29120 external_mini_cluster.cc:1428] Setting key db7e091eb3dbc545b09902bf8d7bb9b5
Aug 09 07:57:22 dist-test-slave-vpvm krb5kdc[26009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262242, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.3@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:57:22.946545 26139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:22.936251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:46035 (local address 127.28.112.62:35083)
0809 07:57:22.936380 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:22.936384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:22.937767 (+ 1383us) server_negotiation.cc:407] Connection header received
0809 07:57:22.938644 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:22.938647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:22.938694 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:22.938757 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:22.939808 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.940340 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.941369 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.941566 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.941632 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:22.943824 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:22.943827 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:22.943829 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:22.943854 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:22.944936 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:22.945355 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:22.945357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:22.945358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:22.945425 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:22.945669 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:22.945671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:22.945672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:22.945826 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:22.945895 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:22.946334 (+ 439us) server_negotiation.cc:299] Negotiation successful
0809 07:57:22.946428 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 07:57:22.947249 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:22.936734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:35083 (local address 127.28.112.3:46035)
0809 07:57:22.937652 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:22.937692 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:22.938435 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:22.938900 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:22.938906 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:22.939234 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:22.939698 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:22.939712 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.940432 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.940434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:22.941278 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:22.941287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.941676 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.941679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:22.941806 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:22.942304 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:22.942319 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:22.943678 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:22.945037 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:22.945042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:22.945052 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:22.945283 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:22.945484 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:22.945485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:22.945487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:22.945607 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:22.945894 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:22.945898 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:22.946143 (+ 245us) client_negotiation.cc:769] Sending connection context
0809 07:57:22.946292 (+ 149us) client_negotiation.cc:240] Negotiation successful
0809 07:57:22.946513 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":112,"threads_started":1}
I20260809 07:57:22.948232 26784 heartbeater.cc:347] Connected to a master server at 127.28.112.62:35083
I20260809 07:57:22.948439 26784 heartbeater.cc:464] Registering TS with master...
I20260809 07:57:22.948938 26784 heartbeater.cc:511] Master 127.28.112.62:35083 requested a full tablet report, sending...
I20260809 07:57:22.950022 26077 ts_manager.cc:194] Registered new tserver with Master: aa156f6ee78f4249b3bad7f6dc10e6b4 (127.28.112.3:36033)
I20260809 07:57:22.950594 26077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.112.3@KRBTEST.COM'} at 127.28.112.3:46035
I20260809 07:57:22.961542 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 07:57:22.970036 26139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:22.963381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.28.112.62:35083)
0809 07:57:22.963488 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:22.963493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:22.963689 (+ 196us) server_negotiation.cc:407] Connection header received
0809 07:57:22.963801 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:22.963804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:22.963848 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:22.963911 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:22.964582 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.965073 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.965869 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.966113 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.966202 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:22.967445 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:22.967448 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:22.967451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:22.967477 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:22.968472 (+ 995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:22.968864 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:22.968868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:22.968870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:22.968946 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:22.969191 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:22.969194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:22.969196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:22.969409 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:22.969526 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:22.969761 (+ 235us) server_negotiation.cc:299] Negotiation successful
0809 07:57:22.969867 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 07:57:22.972611 26077 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:57:22.975890 26077 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:57:22.990054 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:22.984031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.28.112.3:36033)
0809 07:57:22.985022 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:22.985035 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:22.985059 (+ 24us) server_negotiation.cc:407] Connection header received
0809 07:57:22.985154 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:22.985160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:22.985580 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:22.985928 (+ 348us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:22.986118 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.987294 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.988713 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.989330 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.989430 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:22.989583 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:22.989798 (+ 215us) server_negotiation.cc:299] Negotiation successful
0809 07:57:22.989887 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":886,"thread_start_us":151,"threads_started":1}
I20260809 07:57:22.990100 26796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:22.983874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:36033 (local address 127.0.0.1:41574)
0809 07:57:22.984552 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:22.984613 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:22.984740 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:22.985725 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:22.985728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:22.985756 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:22.986018 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:22.986026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.987427 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.987430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:22.988605 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:22.988614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.989489 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.989493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:22.989610 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:22.989653 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 07:57:22.989755 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 07:57:22.989873 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":46,"spinlock_wait_cycles":9984,"thread_start_us":70,"threads_started":1}
I20260809 07:57:22.990105 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:22.983970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:46083 (local address 127.0.0.1:43116)
0809 07:57:22.984552 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:22.984604 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:22.984740 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:22.985827 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:22.985830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:22.985849 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:22.986154 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:22.986159 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.987099 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.987102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:22.988442 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:22.988453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.989668 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.989671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:22.989753 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:22.989778 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 07:57:22.989838 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 07:57:22.989909 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":230,"spinlock_wait_cycles":7296,"thread_start_us":121,"threads_started":1}
I20260809 07:57:22.990386 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:22.984639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43116 (local address 127.28.112.2:46083)
0809 07:57:22.985421 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:22.985431 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:22.985451 (+ 20us) server_negotiation.cc:407] Connection header received
0809 07:57:22.985518 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:22.985522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:22.985681 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:22.986000 (+ 319us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:22.986247 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.986986 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.988595 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.989358 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.989455 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:22.989588 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:22.990167 (+ 579us) server_negotiation.cc:299] Negotiation successful
0809 07:57:22.990259 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":111,"threads_started":1}
I20260809 07:57:22.990372 26799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:22.984177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:38163 (local address 127.0.0.1:42850)
0809 07:57:22.985094 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:22.985106 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:22.985199 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:22.985717 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:22.985724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:22.985757 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:22.985999 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:22.986008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.987495 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.987498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:22.988770 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:22.988781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.989997 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.990000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:22.990091 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:22.990124 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 07:57:22.990192 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 07:57:22.990254 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"mutex_wait_us":66,"spinlock_wait_cycles":11648,"thread_start_us":89,"threads_started":1}
I20260809 07:57:22.990700 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:22.984645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42850 (local address 127.28.112.1:38163)
0809 07:57:22.985053 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:22.985065 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:22.985201 (+ 136us) server_negotiation.cc:407] Connection header received
0809 07:57:22.985308 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:22.985313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:22.985514 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:22.985670 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:22.986135 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.987362 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.988916 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:22.989701 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:22.989792 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:22.989917 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:22.990477 (+ 560us) server_negotiation.cc:299] Negotiation successful
0809 07:57:22.990569 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":108,"threads_started":1}
I20260809 07:57:22.993372 26274 tablet_service.cc:1467] Processing CreateTablet for tablet e50f0b0a366944c7a2c7dbc95006c6c3 (DEFAULT_TABLE table=test-table [id=ffa604944aa84c60ad5126b3f1540094]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:57:22.993418 26476 tablet_service.cc:1467] Processing CreateTablet for tablet e50f0b0a366944c7a2c7dbc95006c6c3 (DEFAULT_TABLE table=test-table [id=ffa604944aa84c60ad5126b3f1540094]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:57:22.993418 26683 tablet_service.cc:1467] Processing CreateTablet for tablet e50f0b0a366944c7a2c7dbc95006c6c3 (DEFAULT_TABLE table=test-table [id=ffa604944aa84c60ad5126b3f1540094]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:57:22.994376 26274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50f0b0a366944c7a2c7dbc95006c6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:22.994436 26476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50f0b0a366944c7a2c7dbc95006c6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:22.994445 26683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50f0b0a366944c7a2c7dbc95006c6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:23.000998 26802 tablet_bootstrap.cc:492] T e50f0b0a366944c7a2c7dbc95006c6c3 P 2f95363e8db34360934249d055d84dfd: Bootstrap starting.
I20260809 07:57:23.002018 26803 tablet_bootstrap.cc:492] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4: Bootstrap starting.
I20260809 07:57:23.002334 26804 tablet_bootstrap.cc:492] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673: Bootstrap starting.
I20260809 07:57:23.003346 26802 tablet_bootstrap.cc:654] T e50f0b0a366944c7a2c7dbc95006c6c3 P 2f95363e8db34360934249d055d84dfd: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:23.004164 26803 tablet_bootstrap.cc:654] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:23.004467 26802 log.cc:824] T e50f0b0a366944c7a2c7dbc95006c6c3 P 2f95363e8db34360934249d055d84dfd: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:23.004638 26804 tablet_bootstrap.cc:654] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:23.005357 26804 log.cc:824] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:23.005357 26803 log.cc:824] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:23.006690 26802 tablet_bootstrap.cc:492] T e50f0b0a366944c7a2c7dbc95006c6c3 P 2f95363e8db34360934249d055d84dfd: No bootstrap required, opened a new log
I20260809 07:57:23.006873 26802 ts_tablet_manager.cc:1397] T e50f0b0a366944c7a2c7dbc95006c6c3 P 2f95363e8db34360934249d055d84dfd: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260809 07:57:23.007237 26803 tablet_bootstrap.cc:492] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4: No bootstrap required, opened a new log
I20260809 07:57:23.007236 26804 tablet_bootstrap.cc:492] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673: No bootstrap required, opened a new log
I20260809 07:57:23.007411 26803 ts_tablet_manager.cc:1397] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 07:57:23.007411 26804 ts_tablet_manager.cc:1397] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 07:57:23.009790 26802 raft_consensus.cc:348] T e50f0b0a366944c7a2c7dbc95006c6c3 P 2f95363e8db34360934249d055d84dfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa156f6ee78f4249b3bad7f6dc10e6b4" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36033 } } peers { permanent_uuid: "dc56db8252cd4aff8a6cb94693087673" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46083 } } peers { permanent_uuid: "2f95363e8db34360934249d055d84dfd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 38163 } }
I20260809 07:57:23.010136 26802 raft_consensus.cc:374] T e50f0b0a366944c7a2c7dbc95006c6c3 P 2f95363e8db34360934249d055d84dfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:23.010196 26802 raft_consensus.cc:729] T e50f0b0a366944c7a2c7dbc95006c6c3 P 2f95363e8db34360934249d055d84dfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f95363e8db34360934249d055d84dfd, State: Initialized, Role: FOLLOWER
I20260809 07:57:23.010654 26802 consensus_queue.cc:260] T e50f0b0a366944c7a2c7dbc95006c6c3 P 2f95363e8db34360934249d055d84dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa156f6ee78f4249b3bad7f6dc10e6b4" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36033 } } peers { permanent_uuid: "dc56db8252cd4aff8a6cb94693087673" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46083 } } peers { permanent_uuid: "2f95363e8db34360934249d055d84dfd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 38163 } }
I20260809 07:57:23.011063 26804 raft_consensus.cc:348] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa156f6ee78f4249b3bad7f6dc10e6b4" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36033 } } peers { permanent_uuid: "dc56db8252cd4aff8a6cb94693087673" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46083 } } peers { permanent_uuid: "2f95363e8db34360934249d055d84dfd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 38163 } }
I20260809 07:57:23.011154 26803 raft_consensus.cc:348] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa156f6ee78f4249b3bad7f6dc10e6b4" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36033 } } peers { permanent_uuid: "dc56db8252cd4aff8a6cb94693087673" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46083 } } peers { permanent_uuid: "2f95363e8db34360934249d055d84dfd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 38163 } }
I20260809 07:57:23.011338 26804 raft_consensus.cc:374] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:23.011350 26803 raft_consensus.cc:374] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:23.011395 26803 raft_consensus.cc:729] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa156f6ee78f4249b3bad7f6dc10e6b4, State: Initialized, Role: FOLLOWER
I20260809 07:57:23.011462 26804 raft_consensus.cc:729] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc56db8252cd4aff8a6cb94693087673, State: Initialized, Role: FOLLOWER
I20260809 07:57:23.011565 26802 ts_tablet_manager.cc:1428] T e50f0b0a366944c7a2c7dbc95006c6c3 P 2f95363e8db34360934249d055d84dfd: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 07:57:23.011911 26374 heartbeater.cc:503] Master 127.28.112.62:35083 was elected leader, sending a full tablet report...
I20260809 07:57:23.011883 26804 consensus_queue.cc:260] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa156f6ee78f4249b3bad7f6dc10e6b4" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36033 } } peers { permanent_uuid: "dc56db8252cd4aff8a6cb94693087673" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46083 } } peers { permanent_uuid: "2f95363e8db34360934249d055d84dfd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 38163 } }
I20260809 07:57:23.011883 26803 consensus_queue.cc:260] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa156f6ee78f4249b3bad7f6dc10e6b4" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36033 } } peers { permanent_uuid: "dc56db8252cd4aff8a6cb94693087673" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46083 } } peers { permanent_uuid: "2f95363e8db34360934249d055d84dfd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 38163 } }
I20260809 07:57:23.012784 26804 ts_tablet_manager.cc:1428] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260809 07:57:23.012784 26803 ts_tablet_manager.cc:1428] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 07:57:23.013074 26582 heartbeater.cc:503] Master 127.28.112.62:35083 was elected leader, sending a full tablet report...
I20260809 07:57:23.013248 26784 heartbeater.cc:503] Master 127.28.112.62:35083 was elected leader, sending a full tablet report...
W20260809 07:57:23.024101 26375 tablet.cc:2369] T e50f0b0a366944c7a2c7dbc95006c6c3 P 2f95363e8db34360934249d055d84dfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 07:57:23.185415 26785 tablet.cc:2369] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 07:57:23.231552 26583 tablet.cc:2369] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:57:23.267149 26811 raft_consensus.cc:482] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:57:23.267349 26811 raft_consensus.cc:504] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa156f6ee78f4249b3bad7f6dc10e6b4" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36033 } } peers { permanent_uuid: "dc56db8252cd4aff8a6cb94693087673" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46083 } } peers { permanent_uuid: "2f95363e8db34360934249d055d84dfd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 38163 } }
I20260809 07:57:23.268553 26811 leader_election.cc:288] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc56db8252cd4aff8a6cb94693087673 (127.28.112.2:46083), 2f95363e8db34360934249d055d84dfd (127.28.112.1:38163)
I20260809 07:57:23.273039 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:23.268799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:46083 (local address 127.28.112.3:34519)
0809 07:57:23.269167 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:23.269182 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:23.269287 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:23.269565 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:23.269568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:23.269584 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:23.269706 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:23.269713 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:23.270731 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:23.270734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:23.271979 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:23.271990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:23.272634 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:23.272638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:23.272741 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:23.272763 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 07:57:23.272830 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 07:57:23.272904 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":59}
I20260809 07:57:23.273170 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:23.269011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:34519 (local address 127.28.112.2:46083)
0809 07:57:23.269150 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:23.269154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:23.269251 (+ 97us) server_negotiation.cc:407] Connection header received
0809 07:57:23.269419 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:23.269423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:23.269476 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:23.269540 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:23.269829 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:23.270610 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:23.272138 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:23.272526 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:23.272640 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:23.272694 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:23.272970 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 07:57:23.273051 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 07:57:23.273298 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:23.269039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:50949 (local address 127.28.112.1:38163)
0809 07:57:23.269175 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:23.269181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:23.269328 (+ 147us) server_negotiation.cc:407] Connection header received
0809 07:57:23.269471 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:23.269474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:23.269525 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:23.269591 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:23.269872 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:23.270422 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:23.272112 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:23.272730 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:23.272844 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:23.272894 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:23.273092 (+ 198us) server_negotiation.cc:299] Negotiation successful
0809 07:57:23.273178 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 07:57:23.273695 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:23.268842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:38163 (local address 127.28.112.3:50949)
0809 07:57:23.269233 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:23.269246 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:23.269355 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:23.269663 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:23.269665 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:23.269680 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:23.269777 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:23.269783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:23.270556 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:23.270559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:23.271979 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:23.271990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:23.272847 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:23.272851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:23.272947 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:23.272970 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 07:57:23.273478 (+ 508us) client_negotiation.cc:240] Negotiation successful
0809 07:57:23.273553 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":42,"thread_start_us":105,"threads_started":1}
I20260809 07:57:23.274080 26506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50f0b0a366944c7a2c7dbc95006c6c3" candidate_uuid: "aa156f6ee78f4249b3bad7f6dc10e6b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc56db8252cd4aff8a6cb94693087673" is_pre_election: true
I20260809 07:57:23.274317 26307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50f0b0a366944c7a2c7dbc95006c6c3" candidate_uuid: "aa156f6ee78f4249b3bad7f6dc10e6b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f95363e8db34360934249d055d84dfd" is_pre_election: true
I20260809 07:57:23.274386 26506 raft_consensus.cc:2454] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa156f6ee78f4249b3bad7f6dc10e6b4 in term 0.
I20260809 07:57:23.274611 26307 raft_consensus.cc:2454] T e50f0b0a366944c7a2c7dbc95006c6c3 P 2f95363e8db34360934249d055d84dfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa156f6ee78f4249b3bad7f6dc10e6b4 in term 0.
I20260809 07:57:23.274866 26617 leader_election.cc:302] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa156f6ee78f4249b3bad7f6dc10e6b4, dc56db8252cd4aff8a6cb94693087673; no voters:
I20260809 07:57:23.275142 26811 raft_consensus.cc:2781] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:57:23.275223 26811 raft_consensus.cc:482] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:57:23.275274 26811 raft_consensus.cc:3037] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:23.276275 26811 raft_consensus.cc:504] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa156f6ee78f4249b3bad7f6dc10e6b4" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36033 } } peers { permanent_uuid: "dc56db8252cd4aff8a6cb94693087673" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46083 } } peers { permanent_uuid: "2f95363e8db34360934249d055d84dfd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 38163 } }
I20260809 07:57:23.276597 26811 leader_election.cc:288] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [CANDIDATE]: Term 1 election: Requested vote from peers dc56db8252cd4aff8a6cb94693087673 (127.28.112.2:46083), 2f95363e8db34360934249d055d84dfd (127.28.112.1:38163)
I20260809 07:57:23.276952 26506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50f0b0a366944c7a2c7dbc95006c6c3" candidate_uuid: "aa156f6ee78f4249b3bad7f6dc10e6b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc56db8252cd4aff8a6cb94693087673"
I20260809 07:57:23.276942 26307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50f0b0a366944c7a2c7dbc95006c6c3" candidate_uuid: "aa156f6ee78f4249b3bad7f6dc10e6b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f95363e8db34360934249d055d84dfd"
I20260809 07:57:23.277057 26307 raft_consensus.cc:3037] T e50f0b0a366944c7a2c7dbc95006c6c3 P 2f95363e8db34360934249d055d84dfd [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:23.277064 26506 raft_consensus.cc:3037] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:23.278067 26506 raft_consensus.cc:2454] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa156f6ee78f4249b3bad7f6dc10e6b4 in term 1.
I20260809 07:57:23.278301 26307 raft_consensus.cc:2454] T e50f0b0a366944c7a2c7dbc95006c6c3 P 2f95363e8db34360934249d055d84dfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa156f6ee78f4249b3bad7f6dc10e6b4 in term 1.
I20260809 07:57:23.278610 26614 leader_election.cc:302] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f95363e8db34360934249d055d84dfd, aa156f6ee78f4249b3bad7f6dc10e6b4; no voters:
I20260809 07:57:23.278792 26811 raft_consensus.cc:2781] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:23.278990 26811 raft_consensus.cc:686] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [term 1 LEADER]: Becoming Leader. State: Replica: aa156f6ee78f4249b3bad7f6dc10e6b4, State: Running, Role: LEADER
I20260809 07:57:23.279275 26811 consensus_queue.cc:237] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa156f6ee78f4249b3bad7f6dc10e6b4" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36033 } } peers { permanent_uuid: "dc56db8252cd4aff8a6cb94693087673" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46083 } } peers { permanent_uuid: "2f95363e8db34360934249d055d84dfd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 38163 } }
I20260809 07:57:23.282198 26075 catalog_manager.cc:5495] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa156f6ee78f4249b3bad7f6dc10e6b4 (127.28.112.3). New cstate: current_term: 1 leader_uuid: "aa156f6ee78f4249b3bad7f6dc10e6b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa156f6ee78f4249b3bad7f6dc10e6b4" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 36033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc56db8252cd4aff8a6cb94693087673" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f95363e8db34360934249d055d84dfd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 38163 } health_report { overall_health: UNKNOWN } } }
I20260809 07:57:23.350244 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:23.347062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.28.112.3:36033)
0809 07:57:23.347200 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:23.347204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:23.347218 (+ 14us) server_negotiation.cc:407] Connection header received
0809 07:57:23.347298 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:23.347300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:23.347356 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:23.347445 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:57:23.347643 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:23.348132 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:23.348941 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:23.349141 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:23.349209 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:23.349449 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:57:23.349878 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:57:23.349988 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:23.350030 (+ 42us) server_negotiation.cc:299] Negotiation successful
0809 07:57:23.350081 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 07:57:23.355906 26506 raft_consensus.cc:1261] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673 [term 1 FOLLOWER]: Refusing update from remote peer aa156f6ee78f4249b3bad7f6dc10e6b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:57:23.355904 26307 raft_consensus.cc:1261] T e50f0b0a366944c7a2c7dbc95006c6c3 P 2f95363e8db34360934249d055d84dfd [term 1 FOLLOWER]: Refusing update from remote peer aa156f6ee78f4249b3bad7f6dc10e6b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:57:23.356642 26815 consensus_queue.cc:1035] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc56db8252cd4aff8a6cb94693087673" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:57:23.356822 26811 consensus_queue.cc:1035] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f95363e8db34360934249d055d84dfd" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 38163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:57:23.366710 26826 mvcc.cc:205] Tried to move back new op lower bound from 7316530148778999808 to 7316530148474228736. Current Snapshot: MvccSnapshot[applied={T|T < 7316530148778999808}]
I20260809 07:57:23.372790 26077 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:57:23.372982 26077 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:57:23.375428 26077 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc69105fcca94ec484bdf7a4343b474d: Sending DeleteTablet for 3 replicas of tablet e50f0b0a366944c7a2c7dbc95006c6c3
I20260809 07:57:23.376163 26683 tablet_service.cc:1514] Processing DeleteTablet for tablet e50f0b0a366944c7a2c7dbc95006c6c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:57:23 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:41574
I20260809 07:57:23.376163 26476 tablet_service.cc:1514] Processing DeleteTablet for tablet e50f0b0a366944c7a2c7dbc95006c6c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:57:23 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:43116
I20260809 07:57:23.376217 26274 tablet_service.cc:1514] Processing DeleteTablet for tablet e50f0b0a366944c7a2c7dbc95006c6c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:57:23 UTC) from {username='kudu', principal='kudu/127.28.112.62@KRBTEST.COM'} at 127.0.0.1:42850
I20260809 07:57:23.376679 26843 tablet_replica.cc:330] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4: stopping tablet replica
I20260809 07:57:23.376839 26844 tablet_replica.cc:330] T e50f0b0a366944c7a2c7dbc95006c6c3 P 2f95363e8db34360934249d055d84dfd: stopping tablet replica
I20260809 07:57:23.377106 26843 raft_consensus.cc:2229] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:57:23.377130 26844 raft_consensus.cc:2229] T e50f0b0a366944c7a2c7dbc95006c6c3 P 2f95363e8db34360934249d055d84dfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:57:23.377442 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 26169
I20260809 07:57:23.377439 26844 raft_consensus.cc:2258] T e50f0b0a366944c7a2c7dbc95006c6c3 P 2f95363e8db34360934249d055d84dfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:57:23.377784 26845 tablet_replica.cc:330] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673: stopping tablet replica
I20260809 07:57:23.380126 26845 raft_consensus.cc:2229] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:57:23.380306 26843 raft_consensus.cc:2258] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:57:23.380487 26845 raft_consensus.cc:2258] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:57:23.383548 26843 ts_tablet_manager.cc:1905] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:57:23.385936 26843 ts_tablet_manager.cc:1918] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:57:23.386116 26843 log.cc:1196] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/e50f0b0a366944c7a2c7dbc95006c6c3
I20260809 07:57:23.386407 26843 ts_tablet_manager.cc:1939] T e50f0b0a366944c7a2c7dbc95006c6c3 P aa156f6ee78f4249b3bad7f6dc10e6b4: Deleting consensus metadata
I20260809 07:57:23.387568 26053 catalog_manager.cc:4841] TS aa156f6ee78f4249b3bad7f6dc10e6b4 (127.28.112.3:36033): tablet e50f0b0a366944c7a2c7dbc95006c6c3 (table test-table [id=ffa604944aa84c60ad5126b3f1540094]) successfully deleted
I20260809 07:57:23.391556 26845 ts_tablet_manager.cc:1905] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 07:57:23.392794 26054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.112.1:38163 (error 108)
I20260809 07:57:23.393867 26799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:23.393568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:38163 (local address 127.0.0.1:42860)
0809 07:57:23.393701 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:23.393774 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260809 07:57:23.393920 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 26379
I20260809 07:57:23.394052 26845 ts_tablet_manager.cc:1918] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:57:23.394164 26845 log.cc:1196] T e50f0b0a366944c7a2c7dbc95006c6c3 P dc56db8252cd4aff8a6cb94693087673: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/e50f0b0a366944c7a2c7dbc95006c6c3
W20260809 07:57:23.398530 26054 catalog_manager.cc:4584] TS 2f95363e8db34360934249d055d84dfd (127.28.112.1:38163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e50f0b0a366944c7a2c7dbc95006c6c3: Network error: Client connection negotiation failed: client connection to 127.28.112.1:38163: connect: Connection refused (error 111)
I20260809 07:57:23.403256 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 26589
I20260809 07:57:23.403329 26799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:23.403073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:46083 (local address 127.0.0.1:43118)
0809 07:57:23.403207 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:23.403257 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.2:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260809 07:57:23.412101 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 26030
2026-08-09T07:57:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2571 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-09T07:57:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:23Z Disabled control of system clock
I20260809 07:57:23.446028 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:33583
--webserver_interface=127.28.112.62
--webserver_port=0
--builtin_ntp_servers=127.28.112.20:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:33583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 07:57:23.553597 26854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:23.553876 26854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:23.553928 26854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:23.558072 26854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:23.558130 26854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:23.558151 26854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:23.558171 26854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:23.558192 26854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:23.562920 26854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:33583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:33583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:23.563944 26854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:23.565291 26854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:23.573256 26861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:23.573261 26863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:23.573272 26860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:23.573417 26854 server_base.cc:1034] running on GCE node
I20260809 07:57:23.574116 26854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:23.575124 26854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:23.576284 26854 hybrid_clock.cc:648] HybridClock initialized: now 1786262243576263 us; error 31 us; skew 500 ppm
I20260809 07:57:23.578048 26854 webserver.cc:460] Webserver started at http://127.28.112.62:40303/ using document root <none> and password file <none>
I20260809 07:57:23.578579 26854 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:23.578648 26854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:23.578848 26854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:23.580818 26854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "3f1e39da912948e19ec4a3a80ad94d8d"
format_stamp: "Formatted at 2026-08-09 07:57:23 on dist-test-slave-vpvm"
server_key: "367ce4d2682aa57c5fc0765a39d91801"
server_key_iv: "baccb29b8865056bcb9e52dbf302a689"
server_key_version: "encryptionkey@0"
I20260809 07:57:23.581337 26854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "3f1e39da912948e19ec4a3a80ad94d8d"
format_stamp: "Formatted at 2026-08-09 07:57:23 on dist-test-slave-vpvm"
server_key: "367ce4d2682aa57c5fc0765a39d91801"
server_key_iv: "baccb29b8865056bcb9e52dbf302a689"
server_key_version: "encryptionkey@0"
I20260809 07:57:23.584803 26854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:57:23.587049 26870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:23.588163 26854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:57:23.588274 26854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "3f1e39da912948e19ec4a3a80ad94d8d"
format_stamp: "Formatted at 2026-08-09 07:57:23 on dist-test-slave-vpvm"
server_key: "367ce4d2682aa57c5fc0765a39d91801"
server_key_iv: "baccb29b8865056bcb9e52dbf302a689"
server_key_version: "encryptionkey@0"
I20260809 07:57:23.588361 26854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:23.605428 26854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:23.606106 26854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:23.606268 26854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:23.612630 26854 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:33583
I20260809 07:57:23.612629 26956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:33583 every 8 connection(s)
I20260809 07:57:23.613557 26854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:23.616714 26957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:23.620857 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 26854
I20260809 07:57:23.620991 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:23.621289 29120 external_mini_cluster.cc:1428] Setting key 1c56cef842008f5675ea5c7013f3322b
I20260809 07:57:23.622675 26957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d: Bootstrap starting.
I20260809 07:57:23.624982 26957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:23.625682 26957 log.cc:824] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:23.627460 26957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d: No bootstrap required, opened a new log
I20260809 07:57:23.630144 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:23.622779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39932 (local address 127.28.112.62:33583)
0809 07:57:23.623411 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:23.623427 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:23.623460 (+ 33us) server_negotiation.cc:407] Connection header received
0809 07:57:23.624066 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:23.624082 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:23.624453 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:23.624762 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:23.625213 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:23.626279 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:23.627123 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:23.627584 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:23.627750 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:23.628504 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:23.628509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:23.628521 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:23.628884 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:23.628937 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:23.628958 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:23.629043 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:23.629185 (+ 142us) server_negotiation.cc:299] Negotiation successful
0809 07:57:23.629425 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":125,"threads_started":1}
I20260809 07:57:23.630894 26957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1e39da912948e19ec4a3a80ad94d8d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 33583 } }
I20260809 07:57:23.631158 26957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:23.631218 26957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1e39da912948e19ec4a3a80ad94d8d, State: Initialized, Role: FOLLOWER
I20260809 07:57:23.631664 26957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1e39da912948e19ec4a3a80ad94d8d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 33583 } }
I20260809 07:57:23.631817 26957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:23.631880 26957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:23.631982 26957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:23.632782 26957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1e39da912948e19ec4a3a80ad94d8d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 33583 } }
I20260809 07:57:23.633103 26957 leader_election.cc:302] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f1e39da912948e19ec4a3a80ad94d8d; no voters:
I20260809 07:57:23.633368 26957 leader_election.cc:288] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:23.633484 26965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:23.633670 26965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d [term 1 LEADER]: Becoming Leader. State: Replica: 3f1e39da912948e19ec4a3a80ad94d8d, State: Running, Role: LEADER
I20260809 07:57:23.634013 26965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1e39da912948e19ec4a3a80ad94d8d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 33583 } }
I20260809 07:57:23.634465 26957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:23.635501 26968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f1e39da912948e19ec4a3a80ad94d8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1e39da912948e19ec4a3a80ad94d8d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 33583 } } }
I20260809 07:57:23.635702 26968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:23.636255 26981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:23.636483 26971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f1e39da912948e19ec4a3a80ad94d8d. Latest consensus state: current_term: 1 leader_uuid: "3f1e39da912948e19ec4a3a80ad94d8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1e39da912948e19ec4a3a80ad94d8d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 33583 } } }
I20260809 07:57:23.636615 26971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:23.639307 26981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:23.644172 26981 catalog_manager.cc:1347] Generated new cluster ID: db77d325de464be9960b26de17dc7f6b
I20260809 07:57:23.644232 26981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:23.650017 26981 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:23.650870 26981 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:23.655699 26981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d: Generated new TSK 0
I20260809 07:57:23.656339 26981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:23.666509 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.1:0
--local_ip_for_outbound_sockets=127.28.112.1
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:33583
--builtin_ntp_servers=127.28.112.20:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 07:57:23.775725 26992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:23.775949 26992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:23.775982 26992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:23.779796 26992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:23.779837 26992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:23.779886 26992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.1
I20260809 07:57:23.784412 26992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.112.1
--webserver_port=0
--tserver_master_addrs=127.28.112.62:33583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.1
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:23.785339 26992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:23.786616 26992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:23.795499 26998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:23.795490 27001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:23.795536 26999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:23.795732 26992 server_base.cc:1034] running on GCE node
I20260809 07:57:23.796234 26992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:23.796821 26992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:23.797989 26992 hybrid_clock.cc:648] HybridClock initialized: now 1786262243797971 us; error 27 us; skew 500 ppm
I20260809 07:57:23.802596 26992 webserver.cc:460] Webserver started at http://127.28.112.1:42613/ using document root <none> and password file <none>
I20260809 07:57:23.803120 26992 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:23.803194 26992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:23.803393 26992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:23.805301 26992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-0/data/instance:
uuid: "51871b9efc424932bd76006c54036d64"
format_stamp: "Formatted at 2026-08-09 07:57:23 on dist-test-slave-vpvm"
server_key: "e6f21f25a4f15e64df8702d2fb0f41ad"
server_key_iv: "a93977fc797ca4f2f5376fad0d6c863e"
server_key_version: "encryptionkey@0"
I20260809 07:57:23.805806 26992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance:
uuid: "51871b9efc424932bd76006c54036d64"
format_stamp: "Formatted at 2026-08-09 07:57:23 on dist-test-slave-vpvm"
server_key: "e6f21f25a4f15e64df8702d2fb0f41ad"
server_key_iv: "a93977fc797ca4f2f5376fad0d6c863e"
server_key_version: "encryptionkey@0"
I20260809 07:57:23.809166 26992 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260809 07:57:23.811697 27011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:23.812812 26992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:57:23.812934 26992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-0/wal
uuid: "51871b9efc424932bd76006c54036d64"
format_stamp: "Formatted at 2026-08-09 07:57:23 on dist-test-slave-vpvm"
server_key: "e6f21f25a4f15e64df8702d2fb0f41ad"
server_key_iv: "a93977fc797ca4f2f5376fad0d6c863e"
server_key_version: "encryptionkey@0"
I20260809 07:57:23.813040 26992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:23.826606 26992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:23.827190 26992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:23.827359 26992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:23.827920 26992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:57:23.828838 26992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:57:23.828889 26992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:23.828949 26992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:57:23.828982 26992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:23.838033 26992 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.1:40401
I20260809 07:57:23.838070 27192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.1:40401 every 8 connection(s)
I20260809 07:57:23.838922 26992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-0/data/info.pb
I20260809 07:57:23.842916 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 26992
I20260809 07:57:23.843047 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-0/wal/instance
I20260809 07:57:23.843384 29120 external_mini_cluster.cc:1428] Setting key ccd8350f8edb744ef5ad28f8d1256b87
I20260809 07:57:23.845345 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.2:0
--local_ip_for_outbound_sockets=127.28.112.2
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:33583
--builtin_ntp_servers=127.28.112.20:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 07:57:23.847781 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:23.840595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.1:35235 (local address 127.28.112.62:33583)
0809 07:57:23.840740 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:23.840746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:23.841763 (+ 1017us) server_negotiation.cc:407] Connection header received
0809 07:57:23.843221 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:23.843225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:23.843277 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:23.843357 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:23.844325 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:23.844917 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:23.846061 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:23.846278 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:23.846367 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:23.847338 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:23.847343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:23.847345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:23.847356 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:23.847377 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:23.847384 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:23.847441 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:23.847575 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 07:57:23.847643 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 07:57:23.848518 27198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:23.840988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:33583 (local address 127.28.112.1:35235)
0809 07:57:23.841615 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:23.841655 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:23.842970 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:23.843545 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:23.843554 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:23.843658 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:23.844177 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:23.844197 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:23.845036 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:23.845039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:23.845927 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:23.845937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:23.846417 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:23.846421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:23.846576 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:23.847103 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:57:23.847105 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:23.847143 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:57:23.847202 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:23.847452 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:23.847461 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:23.847467 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 07:57:23.847567 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 07:57:23.847823 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":123,"threads_started":1}
I20260809 07:57:23.849812 27195 heartbeater.cc:347] Connected to a master server at 127.28.112.62:33583
I20260809 07:57:23.850044 27195 heartbeater.cc:464] Registering TS with master...
I20260809 07:57:23.850510 27195 heartbeater.cc:511] Master 127.28.112.62:33583 requested a full tablet report, sending...
I20260809 07:57:23.851912 26897 ts_manager.cc:194] Registered new tserver with Master: 51871b9efc424932bd76006c54036d64 (127.28.112.1:40401)
I20260809 07:57:23.852994 26897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.112.1:35235
W20260809 07:57:23.954779 27199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:23.955016 27199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:23.955067 27199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:23.958947 27199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:23.958995 27199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:23.959064 27199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.2
I20260809 07:57:23.963703 27199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.112.2
--webserver_port=0
--tserver_master_addrs=127.28.112.62:33583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.2
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:23.964699 27199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:23.966033 27199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:23.975008 27207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:23.975080 27211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:23.975029 27209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:23.975322 27199 server_base.cc:1034] running on GCE node
I20260809 07:57:23.975724 27199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:23.976310 27199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:23.977456 27199 hybrid_clock.cc:648] HybridClock initialized: now 1786262243977441 us; error 31 us; skew 500 ppm
I20260809 07:57:23.979146 27199 webserver.cc:460] Webserver started at http://127.28.112.2:37129/ using document root <none> and password file <none>
I20260809 07:57:23.979681 27199 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:23.979748 27199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:23.979946 27199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:23.981922 27199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-1/data/instance:
uuid: "d8a7a2f051fe4e9989285125f2fa5a91"
format_stamp: "Formatted at 2026-08-09 07:57:23 on dist-test-slave-vpvm"
server_key: "803a58a5d120966f3358114e4752205e"
server_key_iv: "3a14c88ffe26da239cb1392d76a7081b"
server_key_version: "encryptionkey@0"
I20260809 07:57:23.982477 27199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance:
uuid: "d8a7a2f051fe4e9989285125f2fa5a91"
format_stamp: "Formatted at 2026-08-09 07:57:23 on dist-test-slave-vpvm"
server_key: "803a58a5d120966f3358114e4752205e"
server_key_iv: "3a14c88ffe26da239cb1392d76a7081b"
server_key_version: "encryptionkey@0"
I20260809 07:57:23.985888 27199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 07:57:23.988137 27223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:23.989213 27199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 07:57:23.989320 27199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-1/wal
uuid: "d8a7a2f051fe4e9989285125f2fa5a91"
format_stamp: "Formatted at 2026-08-09 07:57:23 on dist-test-slave-vpvm"
server_key: "803a58a5d120966f3358114e4752205e"
server_key_iv: "3a14c88ffe26da239cb1392d76a7081b"
server_key_version: "encryptionkey@0"
I20260809 07:57:23.989409 27199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:24.004531 27199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:24.005136 27199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:24.005299 27199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:24.005837 27199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:57:24.006759 27199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:57:24.006808 27199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:24.006858 27199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:57:24.006887 27199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:24.015195 27199 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.2:39773
I20260809 07:57:24.015214 27399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.2:39773 every 8 connection(s)
I20260809 07:57:24.016073 27199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-1/data/info.pb
I20260809 07:57:24.021931 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 27199
I20260809 07:57:24.022078 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-1/wal/instance
I20260809 07:57:24.022352 29120 external_mini_cluster.cc:1428] Setting key aa10728ffb0abc4519723b646d780a74
I20260809 07:57:24.024259 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.112.3:0
--local_ip_for_outbound_sockets=127.28.112.3
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:33583
--builtin_ntp_servers=127.28.112.20:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 07:57:24.024969 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:24.017831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.2:44739 (local address 127.28.112.62:33583)
0809 07:57:24.018003 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:24.018007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:24.019038 (+ 1031us) server_negotiation.cc:407] Connection header received
0809 07:57:24.020377 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:24.020380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:24.020441 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:24.020518 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:24.021373 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.021920 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.023113 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.023350 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.023442 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:24.024478 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:24.024482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:24.024484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:24.024496 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:24.024517 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:24.024525 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:24.024560 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:24.024761 (+ 201us) server_negotiation.cc:299] Negotiation successful
0809 07:57:24.024827 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 07:57:24.025785 27404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:24.018282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:33583 (local address 127.28.112.2:44739)
0809 07:57:24.018899 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:24.018937 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:24.020173 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:24.020695 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:24.020701 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:24.020783 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:24.021234 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:24.021246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.022063 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.022067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:24.022986 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:24.022996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.023488 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.023493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:24.023657 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:24.024219 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:57:24.024220 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:24.024257 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:57:24.024316 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:24.024612 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:24.024619 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:24.024623 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 07:57:24.024732 (+ 109us) client_negotiation.cc:240] Negotiation successful
0809 07:57:24.024996 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
I20260809 07:57:24.026888 27400 heartbeater.cc:347] Connected to a master server at 127.28.112.62:33583
I20260809 07:57:24.027107 27400 heartbeater.cc:464] Registering TS with master...
I20260809 07:57:24.027578 27400 heartbeater.cc:511] Master 127.28.112.62:33583 requested a full tablet report, sending...
I20260809 07:57:24.028586 26897 ts_manager.cc:194] Registered new tserver with Master: d8a7a2f051fe4e9989285125f2fa5a91 (127.28.112.2:39773)
I20260809 07:57:24.029137 26897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.112.2:44739
W20260809 07:57:24.137012 27405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:24.137257 27405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:24.137293 27405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:24.141299 27405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:24.141348 27405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:24.141409 27405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.112.3
I20260809 07:57:24.146104 27405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.112.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.112.3
--webserver_port=0
--tserver_master_addrs=127.28.112.62:33583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.112.3
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:24.147056 27405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:24.148365 27405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:24.157833 27415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:24.158015 27405 server_base.cc:1034] running on GCE node
W20260809 07:57:24.157863 27412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:24.157864 27417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:24.158532 27405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:24.159125 27405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:24.160283 27405 hybrid_clock.cc:648] HybridClock initialized: now 1786262244160264 us; error 29 us; skew 500 ppm
I20260809 07:57:24.162045 27405 webserver.cc:460] Webserver started at http://127.28.112.3:46093/ using document root <none> and password file <none>
I20260809 07:57:24.162586 27405 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:24.162652 27405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:24.162847 27405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:24.164814 27405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-2/data/instance:
uuid: "abed593c28a44fcd85daec4ab259fd7b"
format_stamp: "Formatted at 2026-08-09 07:57:24 on dist-test-slave-vpvm"
server_key: "e762301eb5b835fa985913fa5571e1cf"
server_key_iv: "f07d2c64ae0603e032859b244078e2c7"
server_key_version: "encryptionkey@0"
I20260809 07:57:24.165319 27405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance:
uuid: "abed593c28a44fcd85daec4ab259fd7b"
format_stamp: "Formatted at 2026-08-09 07:57:24 on dist-test-slave-vpvm"
server_key: "e762301eb5b835fa985913fa5571e1cf"
server_key_iv: "f07d2c64ae0603e032859b244078e2c7"
server_key_version: "encryptionkey@0"
I20260809 07:57:24.168725 27405 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:57:24.171008 27424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:24.172231 27405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:57:24.172336 27405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-2/wal
uuid: "abed593c28a44fcd85daec4ab259fd7b"
format_stamp: "Formatted at 2026-08-09 07:57:24 on dist-test-slave-vpvm"
server_key: "e762301eb5b835fa985913fa5571e1cf"
server_key_iv: "f07d2c64ae0603e032859b244078e2c7"
server_key_version: "encryptionkey@0"
I20260809 07:57:24.172423 27405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:24.193471 27405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:24.194131 27405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:24.194278 27405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:24.194778 27405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 07:57:24.195653 27405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 07:57:24.195699 27405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:24.195737 27405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 07:57:24.195756 27405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:24.203776 27405 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.3:44119
I20260809 07:57:24.203816 27591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.3:44119 every 8 connection(s)
I20260809 07:57:24.204625 27405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-2/data/info.pb
I20260809 07:57:24.211365 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 27405
I20260809 07:57:24.211472 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-2/wal/instance
I20260809 07:57:24.211752 29120 external_mini_cluster.cc:1428] Setting key cd481a349f921fd0b27339d07f5bcbe5
I20260809 07:57:24.213382 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:24.206322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:45373 (local address 127.28.112.62:33583)
0809 07:57:24.206487 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:24.206492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:24.207493 (+ 1001us) server_negotiation.cc:407] Connection header received
0809 07:57:24.208782 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:24.208784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:24.208831 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:24.208904 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:24.209835 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.210389 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.211505 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.211745 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.211845 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:24.212930 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:24.212933 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:24.212935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:24.212946 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:24.212965 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:24.212973 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:24.213029 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:24.213176 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 07:57:24.213241 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 07:57:24.214210 27597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:24.206727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:33583 (local address 127.28.112.3:45373)
0809 07:57:24.207354 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:24.207394 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:24.208607 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:24.209108 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:24.209115 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:24.209198 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:24.209699 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:24.209714 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.210523 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.210527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:24.211381 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:24.211392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.211907 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.211912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:24.212097 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:24.212687 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 07:57:24.212689 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:24.212728 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 07:57:24.212790 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:24.213053 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:24.213061 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:24.213065 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 07:57:24.213167 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 07:57:24.213421 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":116,"threads_started":1}
I20260809 07:57:24.215166 27592 heartbeater.cc:347] Connected to a master server at 127.28.112.62:33583
I20260809 07:57:24.215372 27592 heartbeater.cc:464] Registering TS with master...
I20260809 07:57:24.215832 27592 heartbeater.cc:511] Master 127.28.112.62:33583 requested a full tablet report, sending...
I20260809 07:57:24.216804 26897 ts_manager.cc:194] Registered new tserver with Master: abed593c28a44fcd85daec4ab259fd7b (127.28.112.3:44119)
I20260809 07:57:24.217360 26897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.112.3:45373
I20260809 07:57:24.225262 29120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 07:57:24.230240 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:24.227087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39942 (local address 127.28.112.62:33583)
0809 07:57:24.227201 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:24.227205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:24.227405 (+ 200us) server_negotiation.cc:407] Connection header received
0809 07:57:24.227490 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:24.227493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:24.227546 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:24.227607 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:24.227818 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.228290 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.228996 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.229187 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.229253 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:24.229868 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:24.229871 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:24.229873 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:24.229884 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:24.229902 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:24.229910 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:24.230014 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:24.230080 (+ 66us) server_negotiation.cc:299] Negotiation successful
0809 07:57:24.230123 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 07:57:24.232573 26897 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 07:57:24.235594 26897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 07:57:24.249290 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:24.243287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:40401 (local address 127.0.0.1:54516)
0809 07:57:24.243945 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:24.243994 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:24.244110 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:24.245125 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:24.245128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:24.245142 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:24.245266 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:24.245274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.246301 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.246305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:24.247638 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:24.247650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.248694 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.248698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:24.248805 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:24.248888 (+ 83us) client_negotiation.cc:769] Sending connection context
0809 07:57:24.248988 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 07:57:24.249109 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":67,"thread_start_us":84,"threads_started":1}
I20260809 07:57:24.249301 27612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:24.243436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.28.112.1:40401)
0809 07:57:24.244727 (+ 1291us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:24.244738 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:24.244759 (+ 21us) server_negotiation.cc:407] Connection header received
0809 07:57:24.244828 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:24.244832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:24.244992 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:24.245260 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:24.245394 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.246196 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.247775 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.248559 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.248856 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:24.249019 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:24.249069 (+ 50us) server_negotiation.cc:299] Negotiation successful
0809 07:57:24.249159 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1184,"thread_start_us":137,"threads_started":1}
I20260809 07:57:24.249650 27611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:24.243684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.3:44119 (local address 127.0.0.1:42640)
0809 07:57:24.244393 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:24.244405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:24.244480 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:24.245082 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:24.245086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:24.245105 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:24.245389 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:24.245394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.246687 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.246690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:24.247687 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:24.247696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.248893 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.248897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:24.248972 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:24.248994 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 07:57:24.249079 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 07:57:24.249147 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":21,"thread_start_us":91,"threads_started":1}
I20260809 07:57:24.250034 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:24.243412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:39773 (local address 127.0.0.1:40384)
0809 07:57:24.244153 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:24.244165 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:24.244236 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:24.245029 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:24.245034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:24.245057 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:24.245179 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:24.245186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.246547 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.246551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:24.247672 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:24.247680 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.248801 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.248805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:24.249275 (+ 470us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:24.249298 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 07:57:24.249691 (+ 393us) client_negotiation.cc:240] Negotiation successful
0809 07:57:24.249874 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":2,"thread_start_us":103,"threads_started":1}
I20260809 07:57:24.250097 27615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:24.244171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42640 (local address 127.28.112.3:44119)
0809 07:57:24.244439 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:24.244451 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:24.244475 (+ 24us) server_negotiation.cc:407] Connection header received
0809 07:57:24.244583 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:24.244595 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:24.244795 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:24.244937 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:24.245507 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.246589 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.247819 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.248559 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.248663 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:24.248835 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:24.249814 (+ 979us) server_negotiation.cc:299] Negotiation successful
0809 07:57:24.249912 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":87,"threads_started":1}
I20260809 07:57:24.250828 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:24.244066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.28.112.2:39773)
0809 07:57:24.244557 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:24.244569 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:24.244595 (+ 26us) server_negotiation.cc:407] Connection header received
0809 07:57:24.244671 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:24.244677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:24.244870 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:24.245028 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:24.245291 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.246436 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.247796 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.248409 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.248519 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:24.248729 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:24.250052 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0809 07:57:24.250672 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":118,"threads_started":1}
I20260809 07:57:24.253860 27299 tablet_service.cc:1467] Processing CreateTablet for tablet 448031b9fb9242d8a781ed8819b5b0b4 (DEFAULT_TABLE table=test-table [id=654a031cde8c454da7ce6305e163c2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:57:24.253865 27494 tablet_service.cc:1467] Processing CreateTablet for tablet 448031b9fb9242d8a781ed8819b5b0b4 (DEFAULT_TABLE table=test-table [id=654a031cde8c454da7ce6305e163c2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:57:24.254074 27091 tablet_service.cc:1467] Processing CreateTablet for tablet 448031b9fb9242d8a781ed8819b5b0b4 (DEFAULT_TABLE table=test-table [id=654a031cde8c454da7ce6305e163c2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 07:57:24.255023 27299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 448031b9fb9242d8a781ed8819b5b0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:24.255023 27494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 448031b9fb9242d8a781ed8819b5b0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:24.255275 27091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 448031b9fb9242d8a781ed8819b5b0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:24.261576 27616 tablet_bootstrap.cc:492] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b: Bootstrap starting.
I20260809 07:57:24.263161 27618 tablet_bootstrap.cc:492] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91: Bootstrap starting.
I20260809 07:57:24.263948 27616 tablet_bootstrap.cc:654] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:24.264855 27616 log.cc:824] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:24.265245 27618 tablet_bootstrap.cc:654] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:24.265684 27619 tablet_bootstrap.cc:492] T 448031b9fb9242d8a781ed8819b5b0b4 P 51871b9efc424932bd76006c54036d64: Bootstrap starting.
I20260809 07:57:24.266151 27618 log.cc:824] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:24.266858 27616 tablet_bootstrap.cc:492] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b: No bootstrap required, opened a new log
I20260809 07:57:24.267047 27616 ts_tablet_manager.cc:1397] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 07:57:24.267724 27619 tablet_bootstrap.cc:654] T 448031b9fb9242d8a781ed8819b5b0b4 P 51871b9efc424932bd76006c54036d64: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:24.268132 27618 tablet_bootstrap.cc:492] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91: No bootstrap required, opened a new log
I20260809 07:57:24.268373 27618 ts_tablet_manager.cc:1397] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 07:57:24.268548 27619 log.cc:824] T 448031b9fb9242d8a781ed8819b5b0b4 P 51871b9efc424932bd76006c54036d64: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:24.270622 27619 tablet_bootstrap.cc:492] T 448031b9fb9242d8a781ed8819b5b0b4 P 51871b9efc424932bd76006c54036d64: No bootstrap required, opened a new log
I20260809 07:57:24.270810 27619 ts_tablet_manager.cc:1397] T 448031b9fb9242d8a781ed8819b5b0b4 P 51871b9efc424932bd76006c54036d64: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 07:57:24.270665 27616 raft_consensus.cc:348] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51871b9efc424932bd76006c54036d64" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40401 } } peers { permanent_uuid: "d8a7a2f051fe4e9989285125f2fa5a91" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39773 } } peers { permanent_uuid: "abed593c28a44fcd85daec4ab259fd7b" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44119 } }
I20260809 07:57:24.270892 27616 raft_consensus.cc:374] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:24.270987 27616 raft_consensus.cc:729] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abed593c28a44fcd85daec4ab259fd7b, State: Initialized, Role: FOLLOWER
I20260809 07:57:24.271407 27616 consensus_queue.cc:260] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51871b9efc424932bd76006c54036d64" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40401 } } peers { permanent_uuid: "d8a7a2f051fe4e9989285125f2fa5a91" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39773 } } peers { permanent_uuid: "abed593c28a44fcd85daec4ab259fd7b" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44119 } }
I20260809 07:57:24.272030 27592 heartbeater.cc:503] Master 127.28.112.62:33583 was elected leader, sending a full tablet report...
I20260809 07:57:24.272298 27616 ts_tablet_manager.cc:1428] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 07:57:24.272365 27618 raft_consensus.cc:348] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51871b9efc424932bd76006c54036d64" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40401 } } peers { permanent_uuid: "d8a7a2f051fe4e9989285125f2fa5a91" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39773 } } peers { permanent_uuid: "abed593c28a44fcd85daec4ab259fd7b" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44119 } }
I20260809 07:57:24.272616 27618 raft_consensus.cc:374] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:24.272681 27618 raft_consensus.cc:729] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8a7a2f051fe4e9989285125f2fa5a91, State: Initialized, Role: FOLLOWER
I20260809 07:57:24.273090 27618 consensus_queue.cc:260] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51871b9efc424932bd76006c54036d64" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40401 } } peers { permanent_uuid: "d8a7a2f051fe4e9989285125f2fa5a91" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39773 } } peers { permanent_uuid: "abed593c28a44fcd85daec4ab259fd7b" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44119 } }
I20260809 07:57:24.274035 27618 ts_tablet_manager.cc:1428] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 07:57:24.274117 27400 heartbeater.cc:503] Master 127.28.112.62:33583 was elected leader, sending a full tablet report...
I20260809 07:57:24.275943 27619 raft_consensus.cc:348] T 448031b9fb9242d8a781ed8819b5b0b4 P 51871b9efc424932bd76006c54036d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51871b9efc424932bd76006c54036d64" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40401 } } peers { permanent_uuid: "d8a7a2f051fe4e9989285125f2fa5a91" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39773 } } peers { permanent_uuid: "abed593c28a44fcd85daec4ab259fd7b" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44119 } }
I20260809 07:57:24.276144 27619 raft_consensus.cc:374] T 448031b9fb9242d8a781ed8819b5b0b4 P 51871b9efc424932bd76006c54036d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:24.276226 27619 raft_consensus.cc:729] T 448031b9fb9242d8a781ed8819b5b0b4 P 51871b9efc424932bd76006c54036d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51871b9efc424932bd76006c54036d64, State: Initialized, Role: FOLLOWER
I20260809 07:57:24.276621 27619 consensus_queue.cc:260] T 448031b9fb9242d8a781ed8819b5b0b4 P 51871b9efc424932bd76006c54036d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51871b9efc424932bd76006c54036d64" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40401 } } peers { permanent_uuid: "d8a7a2f051fe4e9989285125f2fa5a91" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39773 } } peers { permanent_uuid: "abed593c28a44fcd85daec4ab259fd7b" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44119 } }
I20260809 07:57:24.277223 27195 heartbeater.cc:503] Master 127.28.112.62:33583 was elected leader, sending a full tablet report...
I20260809 07:57:24.277505 27619 ts_tablet_manager.cc:1428] T 448031b9fb9242d8a781ed8819b5b0b4 P 51871b9efc424932bd76006c54036d64: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260809 07:57:24.278985 27625 raft_consensus.cc:482] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 07:57:24.279148 27625 raft_consensus.cc:504] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51871b9efc424932bd76006c54036d64" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40401 } } peers { permanent_uuid: "d8a7a2f051fe4e9989285125f2fa5a91" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39773 } } peers { permanent_uuid: "abed593c28a44fcd85daec4ab259fd7b" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44119 } }
I20260809 07:57:24.280331 27625 leader_election.cc:288] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51871b9efc424932bd76006c54036d64 (127.28.112.1:40401), d8a7a2f051fe4e9989285125f2fa5a91 (127.28.112.2:39773)
I20260809 07:57:24.284194 27597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:24.280492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:40401 (local address 127.28.112.3:54699)
0809 07:57:24.280591 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:24.280606 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:24.280685 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:24.281114 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:24.281117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:24.281136 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:24.281277 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:24.281284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.281972 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.281976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:24.283064 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:24.283074 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.283800 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.283803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:24.283905 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:24.283933 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 07:57:24.283998 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 07:57:24.284063 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":24}
I20260809 07:57:24.284528 27612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:24.280725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:54699 (local address 127.28.112.1:40401)
0809 07:57:24.280840 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:24.280843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:24.280854 (+ 11us) server_negotiation.cc:407] Connection header received
0809 07:57:24.280890 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:24.280893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:24.280927 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:24.280995 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:24.281381 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.281859 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.283204 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.283673 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.283783 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:24.283833 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:24.284351 (+ 518us) server_negotiation.cc:299] Negotiation successful
0809 07:57:24.284423 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 07:57:24.285035 27630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:24.280492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.2:39773 (local address 127.28.112.3:40103)
0809 07:57:24.280893 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:24.280903 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:24.280978 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:24.281163 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:24.281165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:24.281176 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 07:57:24.281288 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:24.281292 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.282803 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.282806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:24.283727 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:24.283738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.284654 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.284658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:24.284751 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:24.284769 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 07:57:24.284831 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 07:57:24.284894 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":35,"thread_start_us":99,"threads_started":1}
I20260809 07:57:24.285034 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:24.280569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.112.3:40103 (local address 127.28.112.2:39773)
0809 07:57:24.280680 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:24.280685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:24.280971 (+ 286us) server_negotiation.cc:407] Connection header received
0809 07:57:24.281041 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:24.281044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:24.281085 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:24.281153 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 07:57:24.281997 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.282716 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.283859 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.284407 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.284493 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:24.284534 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:24.284866 (+ 332us) server_negotiation.cc:299] Negotiation successful
0809 07:57:24.284937 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 07:57:24.285014 27128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "448031b9fb9242d8a781ed8819b5b0b4" candidate_uuid: "abed593c28a44fcd85daec4ab259fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51871b9efc424932bd76006c54036d64" is_pre_election: true
I20260809 07:57:24.285282 27128 raft_consensus.cc:2454] T 448031b9fb9242d8a781ed8819b5b0b4 P 51871b9efc424932bd76006c54036d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abed593c28a44fcd85daec4ab259fd7b in term 0.
I20260809 07:57:24.285735 27427 leader_election.cc:302] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51871b9efc424932bd76006c54036d64, abed593c28a44fcd85daec4ab259fd7b; no voters:
I20260809 07:57:24.285809 27329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "448031b9fb9242d8a781ed8819b5b0b4" candidate_uuid: "abed593c28a44fcd85daec4ab259fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8a7a2f051fe4e9989285125f2fa5a91" is_pre_election: true
I20260809 07:57:24.286032 27625 raft_consensus.cc:2781] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 07:57:24.286087 27329 raft_consensus.cc:2454] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abed593c28a44fcd85daec4ab259fd7b in term 0.
I20260809 07:57:24.286110 27625 raft_consensus.cc:482] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 07:57:24.286165 27625 raft_consensus.cc:3037] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:24.287142 27625 raft_consensus.cc:504] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51871b9efc424932bd76006c54036d64" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40401 } } peers { permanent_uuid: "d8a7a2f051fe4e9989285125f2fa5a91" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39773 } } peers { permanent_uuid: "abed593c28a44fcd85daec4ab259fd7b" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44119 } }
I20260809 07:57:24.287446 27625 leader_election.cc:288] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [CANDIDATE]: Term 1 election: Requested vote from peers 51871b9efc424932bd76006c54036d64 (127.28.112.1:40401), d8a7a2f051fe4e9989285125f2fa5a91 (127.28.112.2:39773)
I20260809 07:57:24.287772 27128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "448031b9fb9242d8a781ed8819b5b0b4" candidate_uuid: "abed593c28a44fcd85daec4ab259fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51871b9efc424932bd76006c54036d64"
I20260809 07:57:24.287770 27329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "448031b9fb9242d8a781ed8819b5b0b4" candidate_uuid: "abed593c28a44fcd85daec4ab259fd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8a7a2f051fe4e9989285125f2fa5a91"
I20260809 07:57:24.287884 27128 raft_consensus.cc:3037] T 448031b9fb9242d8a781ed8819b5b0b4 P 51871b9efc424932bd76006c54036d64 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:24.287884 27329 raft_consensus.cc:3037] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:24.289060 27329 raft_consensus.cc:2454] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abed593c28a44fcd85daec4ab259fd7b in term 1.
I20260809 07:57:24.289060 27128 raft_consensus.cc:2454] T 448031b9fb9242d8a781ed8819b5b0b4 P 51871b9efc424932bd76006c54036d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abed593c28a44fcd85daec4ab259fd7b in term 1.
I20260809 07:57:24.289336 27427 leader_election.cc:302] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51871b9efc424932bd76006c54036d64, abed593c28a44fcd85daec4ab259fd7b; no voters:
I20260809 07:57:24.289528 27625 raft_consensus.cc:2781] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:24.289718 27625 raft_consensus.cc:686] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [term 1 LEADER]: Becoming Leader. State: Replica: abed593c28a44fcd85daec4ab259fd7b, State: Running, Role: LEADER
I20260809 07:57:24.290022 27625 consensus_queue.cc:237] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51871b9efc424932bd76006c54036d64" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40401 } } peers { permanent_uuid: "d8a7a2f051fe4e9989285125f2fa5a91" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39773 } } peers { permanent_uuid: "abed593c28a44fcd85daec4ab259fd7b" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44119 } }
I20260809 07:57:24.292961 26897 catalog_manager.cc:5495] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b reported cstate change: term changed from 0 to 1, leader changed from <none> to abed593c28a44fcd85daec4ab259fd7b (127.28.112.3). New cstate: current_term: 1 leader_uuid: "abed593c28a44fcd85daec4ab259fd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51871b9efc424932bd76006c54036d64" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8a7a2f051fe4e9989285125f2fa5a91" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abed593c28a44fcd85daec4ab259fd7b" member_type: VOTER last_known_addr { host: "127.28.112.3" port: 44119 } health_report { overall_health: HEALTHY } } }
I20260809 07:57:24.314134 27615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:24.311036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42656 (local address 127.28.112.3:44119)
0809 07:57:24.311173 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:24.311178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:24.311196 (+ 18us) server_negotiation.cc:407] Connection header received
0809 07:57:24.311280 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:24.311282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:24.311351 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:24.311445 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:57:24.311611 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.312096 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.312869 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:24.313091 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:24.313178 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:24.313390 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:57:24.313784 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:57:24.313869 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:24.313898 (+ 29us) server_negotiation.cc:299] Negotiation successful
0809 07:57:24.313940 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 07:57:24.319782 27128 raft_consensus.cc:1261] T 448031b9fb9242d8a781ed8819b5b0b4 P 51871b9efc424932bd76006c54036d64 [term 1 FOLLOWER]: Refusing update from remote peer abed593c28a44fcd85daec4ab259fd7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:57:24.319782 27329 raft_consensus.cc:1261] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91 [term 1 FOLLOWER]: Refusing update from remote peer abed593c28a44fcd85daec4ab259fd7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:57:24.320454 27631 consensus_queue.cc:1035] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8a7a2f051fe4e9989285125f2fa5a91" member_type: VOTER last_known_addr { host: "127.28.112.2" port: 39773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:57:24.320684 27625 consensus_queue.cc:1035] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "51871b9efc424932bd76006c54036d64" member_type: VOTER last_known_addr { host: "127.28.112.1" port: 40401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:57:24.328826 27641 mvcc.cc:205] Tried to move back new op lower bound from 7316530152727105536 to 7316530152614375424. Current Snapshot: MvccSnapshot[applied={T|T < 7316530152727105536}]
I20260809 07:57:24.330469 27638 mvcc.cc:205] Tried to move back new op lower bound from 7316530152727105536 to 7316530152614375424. Current Snapshot: MvccSnapshot[applied={T|T < 7316530152727105536}]
I20260809 07:57:24.336760 26897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:57:24.336905 26897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 07:57:24.339190 26897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f1e39da912948e19ec4a3a80ad94d8d: Sending DeleteTablet for 3 replicas of tablet 448031b9fb9242d8a781ed8819b5b0b4
I20260809 07:57:24.339866 27494 tablet_service.cc:1514] Processing DeleteTablet for tablet 448031b9fb9242d8a781ed8819b5b0b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:57:24 UTC) from {username='slave'} at 127.0.0.1:42640
I20260809 07:57:24.339870 27091 tablet_service.cc:1514] Processing DeleteTablet for tablet 448031b9fb9242d8a781ed8819b5b0b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:57:24 UTC) from {username='slave'} at 127.0.0.1:54516
I20260809 07:57:24.339947 27299 tablet_service.cc:1514] Processing DeleteTablet for tablet 448031b9fb9242d8a781ed8819b5b0b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 07:57:24 UTC) from {username='slave'} at 127.0.0.1:40384
I20260809 07:57:24.340536 27655 tablet_replica.cc:330] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b: stopping tablet replica
I20260809 07:57:24.340821 27655 raft_consensus.cc:2229] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [term 1 LEADER]: Raft consensus shutting down.
I20260809 07:57:24.341125 27655 raft_consensus.cc:2258] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:57:24.341267 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 26992
I20260809 07:57:24.350535 27655 ts_tablet_manager.cc:1905] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:57:24.352720 27655 ts_tablet_manager.cc:1918] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:57:24.352813 27655 log.cc:1196] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-2/wal/wals/448031b9fb9242d8a781ed8819b5b0b4
I20260809 07:57:24.353129 27655 ts_tablet_manager.cc:1939] T 448031b9fb9242d8a781ed8819b5b0b4 P abed593c28a44fcd85daec4ab259fd7b: Deleting consensus metadata
I20260809 07:57:24.354163 27656 tablet_replica.cc:330] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91: stopping tablet replica
I20260809 07:57:24.354409 26877 catalog_manager.cc:4841] TS abed593c28a44fcd85daec4ab259fd7b (127.28.112.3:44119): tablet 448031b9fb9242d8a781ed8819b5b0b4 (table test-table [id=654a031cde8c454da7ce6305e163c2c5]) successfully deleted
I20260809 07:57:24.354532 27656 raft_consensus.cc:2229] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 07:57:24.354847 27656 raft_consensus.cc:2258] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 07:57:24.356259 27656 ts_tablet_manager.cc:1905] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 07:57:24.358815 27656 ts_tablet_manager.cc:1918] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 07:57:24.358908 27656 log.cc:1196] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91: Deleting WAL directory at /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786261991120089-29120-0/minicluster-data/ts-1/wal/wals/448031b9fb9242d8a781ed8819b5b0b4
I20260809 07:57:24.359223 27656 ts_tablet_manager.cc:1939] T 448031b9fb9242d8a781ed8819b5b0b4 P d8a7a2f051fe4e9989285125f2fa5a91: Deleting consensus metadata
W20260809 07:57:24.360046 26876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.112.1:40401 (error 108)
I20260809 07:57:24.360579 26873 catalog_manager.cc:4841] TS d8a7a2f051fe4e9989285125f2fa5a91 (127.28.112.2:39773): tablet 448031b9fb9242d8a781ed8819b5b0b4 (table test-table [id=654a031cde8c454da7ce6305e163c2c5]) successfully deleted
W20260809 07:57:24.361217 27613 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:24.360908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.1:40401 (local address 127.0.0.1:54518)
0809 07:57:24.361037 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:24.361122 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.1:40401: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 07:57:24.361344 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 27199
W20260809 07:57:24.361377 26876 catalog_manager.cc:4584] TS 51871b9efc424932bd76006c54036d64 (127.28.112.1:40401): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 448031b9fb9242d8a781ed8819b5b0b4: Network error: Client connection negotiation failed: client connection to 127.28.112.1:40401: connect: Connection reset by peer (error 104)
I20260809 07:57:24.373227 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 27405
I20260809 07:57:24.383060 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 26854
2026-08-09T07:57:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (968 ms)
[----------] 6 tests from EncryptionPolicyTest (11083 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:24 dist-test-slave-vpvm krb5kdc[27661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:24 dist-test-slave-vpvm krb5kdc[27661](info): set up 2 sockets
Aug 09 07:57:24 dist-test-slave-vpvm krb5kdc[27661](info): commencing operation
krb5kdc: starting...
W20260809 07:57:25.509666 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:25 dist-test-slave-vpvm krb5kdc[27661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 07:57:25.624497 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35309
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:25.733467 27682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:25.733736 27682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:25.733805 27682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:25.737722 27682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:25.737787 27682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:25.737819 27682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:25.737847 27682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:25.737881 27682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:25.742990 27682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35309
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:25.744263 27682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:25.745286 27682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:25.753350 27688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:25.753346 27692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:25.753407 27690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:25.753427 27682 server_base.cc:1034] running on GCE node
I20260809 07:57:25.754134 27682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:25.755156 27682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:25.756309 27682 hybrid_clock.cc:648] HybridClock initialized: now 1786262245756295 us; error 26 us; skew 500 ppm
Aug 09 07:57:25 dist-test-slave-vpvm krb5kdc[27661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:25.758893 27682 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 07:57:25.759716 27682 webserver.cc:460] Webserver started at http://127.0.0.1:33021/ using document root <none> and password file <none>
I20260809 07:57:25.760221 27682 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:25.760279 27682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:25.760475 27682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:25.762430 27682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "d5fc19b37542455788e34b3936790019"
format_stamp: "Formatted at 2026-08-09 07:57:25 on dist-test-slave-vpvm"
server_key: "80991a913af462cd43a88b442a1c244d"
server_key_iv: "6ac4be91662d09fe2a6a89be118b770b"
server_key_version: "encryptionkey@0"
I20260809 07:57:25.762993 27682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "d5fc19b37542455788e34b3936790019"
format_stamp: "Formatted at 2026-08-09 07:57:25 on dist-test-slave-vpvm"
server_key: "80991a913af462cd43a88b442a1c244d"
server_key_iv: "6ac4be91662d09fe2a6a89be118b770b"
server_key_version: "encryptionkey@0"
I20260809 07:57:25.766495 27682 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 07:57:25.768692 27703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:25.769783 27682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:57:25.769889 27682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "d5fc19b37542455788e34b3936790019"
format_stamp: "Formatted at 2026-08-09 07:57:25 on dist-test-slave-vpvm"
server_key: "80991a913af462cd43a88b442a1c244d"
server_key_iv: "6ac4be91662d09fe2a6a89be118b770b"
server_key_version: "encryptionkey@0"
I20260809 07:57:25.770018 27682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:25.793826 27682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:25.797029 27682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:25.797226 27682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:25.804623 27682 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35309
I20260809 07:57:25.804641 27782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35309 every 8 connection(s)
I20260809 07:57:25.805593 27682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:25.808332 27783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:25.810051 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 27682
I20260809 07:57:25.810190 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:25.810491 29120 external_mini_cluster.cc:1428] Setting key aab330bb10de48e76982a16e00360e67
I20260809 07:57:25.814613 27783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019: Bootstrap starting.
I20260809 07:57:25.816776 27783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:25.817468 27783 log.cc:824] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019: Log is configured to *not* fsync() on all Append() calls
Aug 09 07:57:25 dist-test-slave-vpvm krb5kdc[27661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 07:57:25.819514 27783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019: No bootstrap required, opened a new log
I20260809 07:57:25.822888 27783 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5fc19b37542455788e34b3936790019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35309 } }
I20260809 07:57:25.823103 27783 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:25.823179 27783 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5fc19b37542455788e34b3936790019, State: Initialized, Role: FOLLOWER
I20260809 07:57:25.823602 27783 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5fc19b37542455788e34b3936790019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35309 } }
I20260809 07:57:25.823740 27783 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:25.823834 27783 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:25.823961 27783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:25.824748 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:25.811946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41739 (local address 127.0.0.1:35309)
0809 07:57:25.812384 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:25.812403 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:25.812444 (+ 41us) server_negotiation.cc:407] Connection header received
0809 07:57:25.813081 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:25.813098 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:25.813419 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:25.813722 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:25.814615 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:25.815682 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:25.816569 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:25.817072 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:25.817226 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:25.819974 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:25.819980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:25.819993 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:25.820021 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:25.821734 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:25.822309 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:25.822314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:25.822319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:25.822436 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:25.822799 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:25.822802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:25.822803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:25.823235 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:25.823474 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:25.823811 (+ 337us) server_negotiation.cc:299] Negotiation successful
0809 07:57:25.824035 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":140,"threads_started":1}
I20260809 07:57:25.824800 27783 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5fc19b37542455788e34b3936790019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35309 } }
I20260809 07:57:25.825115 27783 leader_election.cc:302] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5fc19b37542455788e34b3936790019; no voters:
I20260809 07:57:25.825397 27783 leader_election.cc:288] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:25.825485 27792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:25.826478 27792 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019 [term 1 LEADER]: Becoming Leader. State: Replica: d5fc19b37542455788e34b3936790019, State: Running, Role: LEADER
I20260809 07:57:25.826678 27783 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:25.826787 27792 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5fc19b37542455788e34b3936790019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35309 } }
I20260809 07:57:25.827896 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:57:25.828727 27794 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5fc19b37542455788e34b3936790019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5fc19b37542455788e34b3936790019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35309 } } }
I20260809 07:57:25.829015 27794 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:25.828920 27795 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5fc19b37542455788e34b3936790019. Latest consensus state: current_term: 1 leader_uuid: "d5fc19b37542455788e34b3936790019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5fc19b37542455788e34b3936790019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35309 } } }
I20260809 07:57:25.829324 27795 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019 [sys.catalog]: This master's current role is: LEADER
W20260809 07:57:25.831653 27812 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:57:25.831724 27812 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:57:25.831832 27815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:25.832525 27815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:25.838223 27815 catalog_manager.cc:1347] Generated new cluster ID: ba7ae2f6402e42b4ba8d40e9a4d40bcc
I20260809 07:57:25.838289 27815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:25.839304 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:25.830127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54651 (local address 127.0.0.1:35309)
0809 07:57:25.830233 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:25.830237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:25.830442 (+ 205us) server_negotiation.cc:407] Connection header received
0809 07:57:25.830621 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:25.830625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:25.830670 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:25.830716 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:25.831602 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:25.832328 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:25.833466 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:25.833798 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:25.833909 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:25.835553 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:25.835558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:25.835561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:25.835600 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:25.837180 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:25.837756 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:25.837760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:25.837762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:25.837858 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:25.838302 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:25.838305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:25.838307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:25.838540 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:25.838664 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:25.839058 (+ 394us) server_negotiation.cc:299] Negotiation successful
0809 07:57:25.839151 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 07:57:25.859968 27815 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:25.860884 27815 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:25.867110 27815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5fc19b37542455788e34b3936790019: Generated new TSK 0
I20260809 07:57:25.867744 27815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:25.874564 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 27682
W20260809 07:57:25.888854 27659 connection.cc:535] client connection to 127.0.0.1:35309 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35309): Connection reset by peer (error 104)
2026-08-09T07:57:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1496 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:25 dist-test-slave-vpvm krb5kdc[27825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:25 dist-test-slave-vpvm krb5kdc[27825](info): set up 2 sockets
Aug 09 07:57:25 dist-test-slave-vpvm krb5kdc[27825](info): commencing operation
krb5kdc: starting...
W20260809 07:57:27.029582 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:27 dist-test-slave-vpvm krb5kdc[27825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 07:57:27.145550 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43559
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:27.254422 27847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:27.254678 27847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:27.254737 27847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:27.258695 27847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:27.258751 27847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:27.258777 27847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:27.258800 27847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:27.258829 27847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:27.264024 27847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43559
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:27.264976 27847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:27.265982 27847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:27.273969 27853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:27.274066 27847 server_base.cc:1034] running on GCE node
W20260809 07:57:27.274017 27856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:27.273921 27854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:27.274724 27847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:27.275803 27847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:27.276959 27847 hybrid_clock.cc:648] HybridClock initialized: now 1786262247276945 us; error 28 us; skew 500 ppm
Aug 09 07:57:27 dist-test-slave-vpvm krb5kdc[27825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:27.279556 27847 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 07:57:27.280408 27847 webserver.cc:460] Webserver started at http://127.0.0.1:42157/ using document root <none> and password file <none>
I20260809 07:57:27.280934 27847 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:27.280999 27847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:27.281224 27847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:27.283248 27847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "b68b6d5fdad249e5ab3daaf6c8754645"
format_stamp: "Formatted at 2026-08-09 07:57:27 on dist-test-slave-vpvm"
server_key: "8cb52224a72b96c3975accf632168f10"
server_key_iv: "95847a458d19222bc935d6f2b369c2c3"
server_key_version: "encryptionkey@0"
I20260809 07:57:27.283787 27847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "b68b6d5fdad249e5ab3daaf6c8754645"
format_stamp: "Formatted at 2026-08-09 07:57:27 on dist-test-slave-vpvm"
server_key: "8cb52224a72b96c3975accf632168f10"
server_key_iv: "95847a458d19222bc935d6f2b369c2c3"
server_key_version: "encryptionkey@0"
I20260809 07:57:27.287380 27847 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:57:27.289675 27867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:27.290908 27847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:57:27.291021 27847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "b68b6d5fdad249e5ab3daaf6c8754645"
format_stamp: "Formatted at 2026-08-09 07:57:27 on dist-test-slave-vpvm"
server_key: "8cb52224a72b96c3975accf632168f10"
server_key_iv: "95847a458d19222bc935d6f2b369c2c3"
server_key_version: "encryptionkey@0"
I20260809 07:57:27.291123 27847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:27.314548 27847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:27.317376 27847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:27.317540 27847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:27.324223 27847 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43559
I20260809 07:57:27.324237 27944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43559 every 8 connection(s)
I20260809 07:57:27.325181 27847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:27.327999 27947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:27.331116 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 27847
I20260809 07:57:27.331259 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:27.331606 29120 external_mini_cluster.cc:1428] Setting key a69f080e8d01bce9bd70e6dc183ca53a
I20260809 07:57:27.334600 27947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645: Bootstrap starting.
I20260809 07:57:27.336800 27947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:27.337514 27947 log.cc:824] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:27.339356 27947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645: No bootstrap required, opened a new log
I20260809 07:57:27.340850 27953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:27.333508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50453 (local address 127.0.0.1:43559)
0809 07:57:27.333984 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:27.334004 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:27.334046 (+ 42us) server_negotiation.cc:407] Connection header received
0809 07:57:27.334661 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:27.334676 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:27.335042 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:27.335363 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:27.335775 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:27.336736 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:27.337620 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:27.338249 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:27.338399 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:27.339233 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:27.339239 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:27.339251 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:27.339651 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:27.339710 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:27.339734 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:27.339847 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:27.340042 (+ 195us) server_negotiation.cc:299] Negotiation successful
0809 07:57:27.340184 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":138,"threads_started":1}
I20260809 07:57:27.343211 27947 raft_consensus.cc:348] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b68b6d5fdad249e5ab3daaf6c8754645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43559 } }
I20260809 07:57:27.343472 27947 raft_consensus.cc:374] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:27.343529 27947 raft_consensus.cc:729] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b68b6d5fdad249e5ab3daaf6c8754645, State: Initialized, Role: FOLLOWER
I20260809 07:57:27.343904 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:57:27.343968 27947 consensus_queue.cc:260] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b68b6d5fdad249e5ab3daaf6c8754645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43559 } }
I20260809 07:57:27.344118 27947 raft_consensus.cc:388] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:27.344179 27947 raft_consensus.cc:482] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:27.344273 27947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:27.345187 27947 raft_consensus.cc:504] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b68b6d5fdad249e5ab3daaf6c8754645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43559 } }
I20260809 07:57:27.345504 27947 leader_election.cc:302] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b68b6d5fdad249e5ab3daaf6c8754645; no voters:
I20260809 07:57:27.345852 27947 leader_election.cc:288] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:27.346022 27961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:27.346271 27961 raft_consensus.cc:686] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645 [term 1 LEADER]: Becoming Leader. State: Replica: b68b6d5fdad249e5ab3daaf6c8754645, State: Running, Role: LEADER
I20260809 07:57:27.346623 27961 consensus_queue.cc:237] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b68b6d5fdad249e5ab3daaf6c8754645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43559 } }
I20260809 07:57:27.347152 27947 sys_catalog.cc:564] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:27.347716 27961 sys_catalog.cc:455] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b68b6d5fdad249e5ab3daaf6c8754645. Latest consensus state: current_term: 1 leader_uuid: "b68b6d5fdad249e5ab3daaf6c8754645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b68b6d5fdad249e5ab3daaf6c8754645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43559 } } }
I20260809 07:57:27.347890 27961 sys_catalog.cc:458] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:27.348134 27966 sys_catalog.cc:455] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b68b6d5fdad249e5ab3daaf6c8754645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b68b6d5fdad249e5ab3daaf6c8754645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43559 } } }
I20260809 07:57:27.348277 27966 sys_catalog.cc:458] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:27.348337 27977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:27.350889 27953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:27.345812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59267 (local address 127.0.0.1:43559)
0809 07:57:27.345901 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:27.345905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:27.346209 (+ 304us) server_negotiation.cc:407] Connection header received
0809 07:57:27.346400 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:27.346403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:27.346446 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:27.346493 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:27.346891 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:27.347641 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:27.348771 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:27.349093 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:27.349191 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:27.350370 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:27.350374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:27.350376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:27.350390 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:27.350427 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:27.350441 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:27.350520 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:27.350732 (+ 212us) server_negotiation.cc:299] Negotiation successful
0809 07:57:27.350769 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 07:57:27.351816 27977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:27.357079 27977 catalog_manager.cc:1347] Generated new cluster ID: dc3292e0cd93474984664db32173fb06
I20260809 07:57:27.357138 27977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:27.369799 27977 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:27.370666 27977 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:27.376575 27977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b68b6d5fdad249e5ab3daaf6c8754645: Generated new TSK 0
I20260809 07:57:27.377197 27977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:27.385010 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 27847
W20260809 07:57:27.402799 27823 connection.cc:535] client connection to 127.0.0.1:43559 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43559): Connection reset by peer (error 104)
2026-08-09T07:57:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1514 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:27 dist-test-slave-vpvm krb5kdc[28001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:27 dist-test-slave-vpvm krb5kdc[28001](info): set up 2 sockets
Aug 09 07:57:27 dist-test-slave-vpvm krb5kdc[28001](info): commencing operation
krb5kdc: starting...
W20260809 07:57:28.532490 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:28 dist-test-slave-vpvm krb5kdc[28001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 07:57:28.650090 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:28.759343 28018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:28.759620 28018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:28.759680 28018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:28.763629 28018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:28.763684 28018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:28.763710 28018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:28.763733 28018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:28.763762 28018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:28.768908 28018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36583
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:28.769883 28018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:28.770892 28018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:28.779132 28024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:28.779189 28025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:28.779146 28029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:28.779194 28018 server_base.cc:1034] running on GCE node
I20260809 07:57:28.779901 28018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:28.780994 28018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:28.782156 28018 hybrid_clock.cc:648] HybridClock initialized: now 1786262248782138 us; error 31 us; skew 500 ppm
Aug 09 07:57:28 dist-test-slave-vpvm krb5kdc[28001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:28.784739 28018 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 07:57:28.785604 28018 webserver.cc:460] Webserver started at http://127.0.0.1:38301/ using document root <none> and password file <none>
I20260809 07:57:28.786159 28018 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:28.786227 28018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:28.786433 28018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:28.788379 28018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "ddc42673f51547e2971773813abadfe5"
format_stamp: "Formatted at 2026-08-09 07:57:28 on dist-test-slave-vpvm"
server_key: "636f4f7d92d631c757c4f98d29b7683d"
server_key_iv: "0f20c124f175d885a49c28d3aa35ff0f"
server_key_version: "encryptionkey@0"
I20260809 07:57:28.788904 28018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "ddc42673f51547e2971773813abadfe5"
format_stamp: "Formatted at 2026-08-09 07:57:28 on dist-test-slave-vpvm"
server_key: "636f4f7d92d631c757c4f98d29b7683d"
server_key_iv: "0f20c124f175d885a49c28d3aa35ff0f"
server_key_version: "encryptionkey@0"
I20260809 07:57:28.792371 28018 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:57:28.794557 28038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:28.795681 28018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:57:28.795795 28018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "ddc42673f51547e2971773813abadfe5"
format_stamp: "Formatted at 2026-08-09 07:57:28 on dist-test-slave-vpvm"
server_key: "636f4f7d92d631c757c4f98d29b7683d"
server_key_iv: "0f20c124f175d885a49c28d3aa35ff0f"
server_key_version: "encryptionkey@0"
I20260809 07:57:28.795881 28018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:28.800209 28018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:28.806485 28018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:28.806651 28018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:28.813266 28018 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36583
I20260809 07:57:28.813280 28118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36583 every 8 connection(s)
I20260809 07:57:28.814219 28018 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:28.815169 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 28018
I20260809 07:57:28.815325 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:28.815652 29120 external_mini_cluster.cc:1428] Setting key 49456557b8fc1bed7deed3a7039d4217
I20260809 07:57:28.817478 28119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:28.820649 28123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:28.817024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40457 (local address 127.0.0.1:36583)
0809 07:57:28.817465 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:28.817480 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:28.817517 (+ 37us) server_negotiation.cc:407] Connection header received
0809 07:57:28.818221 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:28.818239 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:28.818521 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:28.818821 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:28.819551 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:28.819555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:28.819567 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:28.819578 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:28.819630 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:28.819649 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:28.819775 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:28.819844 (+ 69us) server_negotiation.cc:299] Negotiation successful
0809 07:57:28.819991 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":137,"threads_started":1}
I20260809 07:57:28.823493 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:57:28.823573 28119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5: Bootstrap starting.
I20260809 07:57:28.825930 28119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:28.826821 28123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:28.825395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51727 (local address 127.0.0.1:36583)
0809 07:57:28.825505 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:28.825510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:28.825659 (+ 149us) server_negotiation.cc:407] Connection header received
0809 07:57:28.825740 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:28.825742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:28.825779 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:28.825862 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:28.826488 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:28.826491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:28.826494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:28.826507 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:28.826535 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:28.826547 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:28.826644 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:28.826675 (+ 31us) server_negotiation.cc:299] Negotiation successful
0809 07:57:28.826711 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 07:57:28.826835 28119 log.cc:824] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:28.828754 28119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5: No bootstrap required, opened a new log
I20260809 07:57:28.832384 28119 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc42673f51547e2971773813abadfe5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36583 } }
I20260809 07:57:28.832594 28119 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:28.832640 28119 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddc42673f51547e2971773813abadfe5, State: Initialized, Role: FOLLOWER
I20260809 07:57:28.833027 28119 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc42673f51547e2971773813abadfe5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36583 } }
I20260809 07:57:28.833155 28119 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:28.833218 28119 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:28.833314 28119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:28.834170 28119 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc42673f51547e2971773813abadfe5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36583 } }
I20260809 07:57:28.834489 28119 leader_election.cc:302] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddc42673f51547e2971773813abadfe5; no voters:
I20260809 07:57:28.834760 28119 leader_election.cc:288] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:28.834956 28134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:28.835212 28134 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5 [term 1 LEADER]: Becoming Leader. State: Replica: ddc42673f51547e2971773813abadfe5, State: Running, Role: LEADER
I20260809 07:57:28.835506 28134 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc42673f51547e2971773813abadfe5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36583 } }
I20260809 07:57:28.835897 28119 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:28.837013 28136 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddc42673f51547e2971773813abadfe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc42673f51547e2971773813abadfe5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36583 } } }
I20260809 07:57:28.837146 28136 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:28.837522 28148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:28.837917 28137 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddc42673f51547e2971773813abadfe5. Latest consensus state: current_term: 1 leader_uuid: "ddc42673f51547e2971773813abadfe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddc42673f51547e2971773813abadfe5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36583 } } }
I20260809 07:57:28.838111 28137 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:28.840629 28148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:28.845691 28148 catalog_manager.cc:1347] Generated new cluster ID: 2c8bc5d3cb42431881edb60f1cf2a50b
I20260809 07:57:28.845753 28148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:28.867089 28148 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:28.867520 28148 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:28.877583 28148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddc42673f51547e2971773813abadfe5: Generated new TSK 0
I20260809 07:57:28.878537 28148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:28.894537 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 28018
2026-08-09T07:57:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1504 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:28 dist-test-slave-vpvm krb5kdc[28166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:28 dist-test-slave-vpvm krb5kdc[28166](info): set up 2 sockets
Aug 09 07:57:28 dist-test-slave-vpvm krb5kdc[28166](info): commencing operation
krb5kdc: starting...
W20260809 07:57:30.037163 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:30 dist-test-slave-vpvm krb5kdc[28166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 07:57:30.154673 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42297
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:30.265275 28182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:30.265511 28182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:30.265549 28182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:30.269403 28182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:30.269456 28182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:30.269480 28182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:30.269500 28182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:30.269515 28182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:30.274529 28182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:30.275425 28182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:30.276340 28182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 07:57:30.284255 28182 server_base.cc:1034] running on GCE node
W20260809 07:57:30.284294 28192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:30.284227 28189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:30.284391 28187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:30.285002 28182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:30.286059 28182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:30.287216 28182 hybrid_clock.cc:648] HybridClock initialized: now 1786262250287203 us; error 25 us; skew 500 ppm
Aug 09 07:57:30 dist-test-slave-vpvm krb5kdc[28166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:30.289682 28182 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 07:57:30.290524 28182 webserver.cc:460] Webserver started at http://127.0.0.1:41071/ using document root <none> and password file <none>
I20260809 07:57:30.291003 28182 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:30.291057 28182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:30.291225 28182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:30.293073 28182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "80ecc6b022a44cb49bf1a3bc78e204d4"
format_stamp: "Formatted at 2026-08-09 07:57:30 on dist-test-slave-vpvm"
server_key: "0fc1afdc15676555dd18cebb4e4376ef"
server_key_iv: "e60300e1df3fa6d4a38b50fa5cc65a28"
server_key_version: "encryptionkey@0"
I20260809 07:57:30.293570 28182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "80ecc6b022a44cb49bf1a3bc78e204d4"
format_stamp: "Formatted at 2026-08-09 07:57:30 on dist-test-slave-vpvm"
server_key: "0fc1afdc15676555dd18cebb4e4376ef"
server_key_iv: "e60300e1df3fa6d4a38b50fa5cc65a28"
server_key_version: "encryptionkey@0"
I20260809 07:57:30.296962 28182 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:57:30.299175 28200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:30.300350 28182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:57:30.300472 28182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "80ecc6b022a44cb49bf1a3bc78e204d4"
format_stamp: "Formatted at 2026-08-09 07:57:30 on dist-test-slave-vpvm"
server_key: "0fc1afdc15676555dd18cebb4e4376ef"
server_key_iv: "e60300e1df3fa6d4a38b50fa5cc65a28"
server_key_version: "encryptionkey@0"
I20260809 07:57:30.300559 28182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:30.321362 28182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:30.324021 28182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:30.324200 28182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:30.330778 28182 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42297
I20260809 07:57:30.330821 28272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42297 every 8 connection(s)
I20260809 07:57:30.331701 28182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:30.334455 28276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:30.340034 28276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4: Bootstrap starting.
I20260809 07:57:30.340376 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 28182
I20260809 07:57:30.340517 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:30.340821 29120 external_mini_cluster.cc:1428] Setting key 25eb85f63f4d4f7ff732e49164695cc5
I20260809 07:57:30.342933 28276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:30.343797 28276 log.cc:824] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:30.345620 28276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4: No bootstrap required, opened a new log
I20260809 07:57:30.349192 28276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ecc6b022a44cb49bf1a3bc78e204d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42297 } }
I20260809 07:57:30.349408 28276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:30.349505 28276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ecc6b022a44cb49bf1a3bc78e204d4, State: Initialized, Role: FOLLOWER
I20260809 07:57:30.349946 28276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ecc6b022a44cb49bf1a3bc78e204d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42297 } }
I20260809 07:57:30.350126 28276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:30.350193 28276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:30.350292 28276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:30.351142 28276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ecc6b022a44cb49bf1a3bc78e204d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42297 } }
I20260809 07:57:30.351466 28276 leader_election.cc:302] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80ecc6b022a44cb49bf1a3bc78e204d4; no voters:
I20260809 07:57:30.351752 28276 leader_election.cc:288] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:30.351831 28282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:30.352041 28282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4 [term 1 LEADER]: Becoming Leader. State: Replica: 80ecc6b022a44cb49bf1a3bc78e204d4, State: Running, Role: LEADER
I20260809 07:57:30.352397 28282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ecc6b022a44cb49bf1a3bc78e204d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42297 } }
I20260809 07:57:30.353081 28276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4 [sys.catalog]: configured and running, proceeding with master startup.
Aug 09 07:57:30 dist-test-slave-vpvm krb5kdc[28166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 07:57:30.353636 28282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80ecc6b022a44cb49bf1a3bc78e204d4. Latest consensus state: current_term: 1 leader_uuid: "80ecc6b022a44cb49bf1a3bc78e204d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ecc6b022a44cb49bf1a3bc78e204d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42297 } } }
I20260809 07:57:30.353734 28282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:30.354247 28290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:30.354736 28283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80ecc6b022a44cb49bf1a3bc78e204d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ecc6b022a44cb49bf1a3bc78e204d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42297 } } }
I20260809 07:57:30.354888 28283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:30.357360 28290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:30.360697 28279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:30.342187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42445 (local address 127.0.0.1:42297)
0809 07:57:30.342656 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:30.342677 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:30.342720 (+ 43us) server_negotiation.cc:407] Connection header received
0809 07:57:30.343464 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:30.343485 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:30.343837 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:30.344163 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:30.344986 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:30.346035 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:30.346851 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:30.347311 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:30.347457 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:30.354307 (+ 6850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:30.354314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:30.354332 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:30.354363 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:30.357066 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:30.357615 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:30.357622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:30.357628 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:30.357769 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:30.358331 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:30.358335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:30.358336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:30.358857 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:30.359044 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:30.359511 (+ 467us) server_negotiation.cc:299] Negotiation successful
0809 07:57:30.359799 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":131,"threads_started":1}
I20260809 07:57:30.362844 28290 catalog_manager.cc:1347] Generated new cluster ID: 3ecdd56c3e4d40079a3f51328486c224
I20260809 07:57:30.362952 28290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:30.363933 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:57:30.376590 28279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:30.366394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38861 (local address 127.0.0.1:42297)
0809 07:57:30.366495 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:30.366499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:30.370211 (+ 3712us) server_negotiation.cc:407] Connection header received
0809 07:57:30.370315 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:30.370318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:30.370363 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:30.370427 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:30.371053 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:30.371545 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:30.372306 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:30.372547 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:30.372636 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:30.373842 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:30.373845 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:30.373848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:30.373872 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:30.375139 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:30.375510 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:30.375514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:30.375515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:30.375592 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:30.375832 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:30.375835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:30.375837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:30.376009 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:30.376123 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:30.376337 (+ 214us) server_negotiation.cc:299] Negotiation successful
0809 07:57:30.376440 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 07:57:30.377664 28290 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:30.378806 28290 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:30.393579 28290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80ecc6b022a44cb49bf1a3bc78e204d4: Generated new TSK 0
I20260809 07:57:30.394441 28290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:30.401161 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 28182
2026-08-09T07:57:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1507 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:30 dist-test-slave-vpvm krb5kdc[28318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:30 dist-test-slave-vpvm krb5kdc[28318](info): set up 2 sockets
Aug 09 07:57:30 dist-test-slave-vpvm krb5kdc[28318](info): commencing operation
krb5kdc: starting...
W20260809 07:57:31.534605 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:31 dist-test-slave-vpvm krb5kdc[28318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 07:57:31.654821 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43167
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:31.764802 28334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:31.765079 28334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:31.765120 28334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:31.769061 28334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:31.769119 28334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:31.769136 28334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:31.769153 28334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:31.769181 28334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:31.774358 28334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:31.775381 28334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:31.776371 28334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:31.784516 28341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:31.784596 28345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:31.784623 28334 server_base.cc:1034] running on GCE node
W20260809 07:57:31.784510 28343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:31.785274 28334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:31.786309 28334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:31.787473 28334 hybrid_clock.cc:648] HybridClock initialized: now 1786262251787447 us; error 38 us; skew 500 ppm
Aug 09 07:57:31 dist-test-slave-vpvm krb5kdc[28318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:31.790103 28334 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 07:57:31.790915 28334 webserver.cc:460] Webserver started at http://127.0.0.1:36021/ using document root <none> and password file <none>
I20260809 07:57:31.791429 28334 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:31.791494 28334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:31.791695 28334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:31.793615 28334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "0cf873cf0b684beb8010cb918a19cd53"
format_stamp: "Formatted at 2026-08-09 07:57:31 on dist-test-slave-vpvm"
server_key: "f5d48beab221b1cbf43fa0f321d46986"
server_key_iv: "838598b16b153f4f8ba23ad25d29ce43"
server_key_version: "encryptionkey@0"
I20260809 07:57:31.794199 28334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "0cf873cf0b684beb8010cb918a19cd53"
format_stamp: "Formatted at 2026-08-09 07:57:31 on dist-test-slave-vpvm"
server_key: "f5d48beab221b1cbf43fa0f321d46986"
server_key_iv: "838598b16b153f4f8ba23ad25d29ce43"
server_key_version: "encryptionkey@0"
I20260809 07:57:31.797611 28334 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260809 07:57:31.799890 28354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:31.801002 28334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:57:31.801117 28334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "0cf873cf0b684beb8010cb918a19cd53"
format_stamp: "Formatted at 2026-08-09 07:57:31 on dist-test-slave-vpvm"
server_key: "f5d48beab221b1cbf43fa0f321d46986"
server_key_iv: "838598b16b153f4f8ba23ad25d29ce43"
server_key_version: "encryptionkey@0"
I20260809 07:57:31.801216 28334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:31.820792 28334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:31.823967 28334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:31.824156 28334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:31.830974 28334 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43167
I20260809 07:57:31.830989 28435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43167 every 8 connection(s)
I20260809 07:57:31.831913 28334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:31.834812 28436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:31.840220 28436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53: Bootstrap starting.
I20260809 07:57:31.840405 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 28334
I20260809 07:57:31.840531 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:31.840795 29120 external_mini_cluster.cc:1428] Setting key dffea1c0980b9be1de158ad90bfe43ac
I20260809 07:57:31.842837 28436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:31.843719 28436 log.cc:824] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:31.845573 28436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53: No bootstrap required, opened a new log
I20260809 07:57:31.849603 28436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf873cf0b684beb8010cb918a19cd53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } }
I20260809 07:57:31.849835 28436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:31.849866 28440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:31.842164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51513 (local address 127.0.0.1:43167)
0809 07:57:31.842570 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:31.842599 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:31.842642 (+ 43us) server_negotiation.cc:407] Connection header received
0809 07:57:31.843477 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:31.843502 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:31.843944 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:31.844302 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:31.844711 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:31.845637 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:31.846837 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:31.847301 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:31.847417 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:31.848238 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:31.848243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:31.848256 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:31.848637 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:31.848693 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:31.848716 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:31.848790 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:31.848927 (+ 137us) server_negotiation.cc:299] Negotiation successful
0809 07:57:31.849116 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
I20260809 07:57:31.849895 28436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf873cf0b684beb8010cb918a19cd53, State: Initialized, Role: FOLLOWER
I20260809 07:57:31.850385 28436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf873cf0b684beb8010cb918a19cd53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } }
I20260809 07:57:31.850553 28436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:31.850617 28436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:31.850720 28436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:31.851612 28436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf873cf0b684beb8010cb918a19cd53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } }
I20260809 07:57:31.851970 28436 leader_election.cc:302] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cf873cf0b684beb8010cb918a19cd53; no voters:
I20260809 07:57:31.852269 28436 leader_election.cc:288] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:31.852365 28442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:31.852571 28442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53 [term 1 LEADER]: Becoming Leader. State: Replica: 0cf873cf0b684beb8010cb918a19cd53, State: Running, Role: LEADER
I20260809 07:57:31.852710 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:57:31.852941 28442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf873cf0b684beb8010cb918a19cd53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } }
I20260809 07:57:31.853566 28436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:31.854369 28445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf873cf0b684beb8010cb918a19cd53. Latest consensus state: current_term: 1 leader_uuid: "0cf873cf0b684beb8010cb918a19cd53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf873cf0b684beb8010cb918a19cd53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } } }
I20260809 07:57:31.854497 28445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:31.854802 28443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cf873cf0b684beb8010cb918a19cd53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf873cf0b684beb8010cb918a19cd53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } } }
I20260809 07:57:31.854879 28443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:31.855252 28463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:31.858752 28463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:31.862767 28440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:31.855113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41675 (local address 127.0.0.1:43167)
0809 07:57:31.855227 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:31.855231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:31.858194 (+ 2963us) server_negotiation.cc:407] Connection header received
0809 07:57:31.858323 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:31.858326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:31.858371 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:31.858437 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:31.859166 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:31.859909 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:31.861067 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:31.861374 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:31.861476 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:31.862363 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:31.862367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:31.862370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:31.862383 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:31.862410 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:31.862422 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:31.862497 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:31.862585 (+ 88us) server_negotiation.cc:299] Negotiation successful
0809 07:57:31.862645 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 07:57:31.864699 28463 catalog_manager.cc:1347] Generated new cluster ID: e17c8778522f4867a5f64f3a3161a31c
I20260809 07:57:31.864760 28463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:31.880373 28463 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:31.881646 28463 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:31.895612 28463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cf873cf0b684beb8010cb918a19cd53: Generated new TSK 0
I20260809 07:57:31.896438 28463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:31.905761 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 28334
2026-08-09T07:57:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1510 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:31 dist-test-slave-vpvm krb5kdc[28480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:31 dist-test-slave-vpvm krb5kdc[28480](info): set up 2 sockets
Aug 09 07:57:31 dist-test-slave-vpvm krb5kdc[28480](info): commencing operation
krb5kdc: starting...
W20260809 07:57:33.040040 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:33 dist-test-slave-vpvm krb5kdc[28480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 07:57:33.156406 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:33.266395 28498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:33.266657 28498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:33.266712 28498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:33.270604 28498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:33.270663 28498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:33.270689 28498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:33.270710 28498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:33.270735 28498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:33.275856 28498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:33.276861 28498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:33.277853 28498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:33.285938 28506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:33.285988 28504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:33.286098 28498 server_base.cc:1034] running on GCE node
W20260809 07:57:33.286311 28514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:33.286836 28498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:33.287849 28498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:33.289003 28498 hybrid_clock.cc:648] HybridClock initialized: now 1786262253288986 us; error 28 us; skew 500 ppm
Aug 09 07:57:33 dist-test-slave-vpvm krb5kdc[28480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:33.291586 28498 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 07:57:33.292389 28498 webserver.cc:460] Webserver started at http://127.0.0.1:36895/ using document root <none> and password file <none>
I20260809 07:57:33.292888 28498 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:33.292945 28498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:33.293100 28498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:33.295014 28498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "5f2470b0ed3e4938a7aabb8169b240c5"
format_stamp: "Formatted at 2026-08-09 07:57:33 on dist-test-slave-vpvm"
server_key: "3ef1e3a3d81698db1bdb26c20f71a17f"
server_key_iv: "bc898b2646d5eddd9d4a1f36656764ec"
server_key_version: "encryptionkey@0"
I20260809 07:57:33.295509 28498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "5f2470b0ed3e4938a7aabb8169b240c5"
format_stamp: "Formatted at 2026-08-09 07:57:33 on dist-test-slave-vpvm"
server_key: "3ef1e3a3d81698db1bdb26c20f71a17f"
server_key_iv: "bc898b2646d5eddd9d4a1f36656764ec"
server_key_version: "encryptionkey@0"
I20260809 07:57:33.298957 28498 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:57:33.301237 28524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:33.302404 28498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:57:33.302517 28498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "5f2470b0ed3e4938a7aabb8169b240c5"
format_stamp: "Formatted at 2026-08-09 07:57:33 on dist-test-slave-vpvm"
server_key: "3ef1e3a3d81698db1bdb26c20f71a17f"
server_key_iv: "bc898b2646d5eddd9d4a1f36656764ec"
server_key_version: "encryptionkey@0"
I20260809 07:57:33.302616 28498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:33.306941 28498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:33.309670 28498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:33.309823 28498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:33.316195 28498 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36115
I20260809 07:57:33.316247 28602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36115 every 8 connection(s)
I20260809 07:57:33.317063 28498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:33.319800 28603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:33.322445 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 28498
I20260809 07:57:33.322590 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:33.322911 29120 external_mini_cluster.cc:1428] Setting key 14dbc989f23cb2f131f10ce8255b8b55
I20260809 07:57:33.325646 28603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5: Bootstrap starting.
I20260809 07:57:33.327708 28606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:33.324190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43019 (local address 127.0.0.1:36115)
0809 07:57:33.324597 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:33.324611 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:33.324647 (+ 36us) server_negotiation.cc:407] Connection header received
0809 07:57:33.325229 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:33.325245 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:33.325509 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:33.325804 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:33.326564 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:33.326569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:33.326580 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:33.326596 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:33.326646 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:33.326665 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:33.326764 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:33.326876 (+ 112us) server_negotiation.cc:299] Negotiation successful
0809 07:57:33.327013 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":122,"threads_started":1}
I20260809 07:57:33.328086 28603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:33.328866 28603 log.cc:824] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:33.330461 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:57:33.330832 28603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5: No bootstrap required, opened a new log
I20260809 07:57:33.333750 28606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:33.332371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36197 (local address 127.0.0.1:36115)
0809 07:57:33.332474 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:33.332478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:33.332613 (+ 135us) server_negotiation.cc:407] Connection header received
0809 07:57:33.332725 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:33.332727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:33.332755 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:33.332812 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:33.333430 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:33.333433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:33.333435 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:33.333444 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:33.333466 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:33.333475 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:33.333535 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:33.333630 (+ 95us) server_negotiation.cc:299] Negotiation successful
0809 07:57:33.333658 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 07:57:33.334720 28603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2470b0ed3e4938a7aabb8169b240c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260809 07:57:33.334931 28603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:33.334987 28603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f2470b0ed3e4938a7aabb8169b240c5, State: Initialized, Role: FOLLOWER
I20260809 07:57:33.335386 28603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2470b0ed3e4938a7aabb8169b240c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260809 07:57:33.335511 28603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:33.335577 28603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:33.335673 28603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:33.336510 28603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2470b0ed3e4938a7aabb8169b240c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260809 07:57:33.336854 28603 leader_election.cc:302] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f2470b0ed3e4938a7aabb8169b240c5; no voters:
I20260809 07:57:33.337251 28603 leader_election.cc:288] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:33.337338 28616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:33.337527 28616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5 [term 1 LEADER]: Becoming Leader. State: Replica: 5f2470b0ed3e4938a7aabb8169b240c5, State: Running, Role: LEADER
I20260809 07:57:33.337847 28616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2470b0ed3e4938a7aabb8169b240c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260809 07:57:33.338480 28603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:33.339576 28620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f2470b0ed3e4938a7aabb8169b240c5. Latest consensus state: current_term: 1 leader_uuid: "5f2470b0ed3e4938a7aabb8169b240c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2470b0ed3e4938a7aabb8169b240c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } } }
I20260809 07:57:33.339695 28620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:33.339897 28619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f2470b0ed3e4938a7aabb8169b240c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2470b0ed3e4938a7aabb8169b240c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } } }
I20260809 07:57:33.340022 28619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5 [sys.catalog]: This master's current role is: LEADER
W20260809 07:57:33.343124 28641 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:57:33.343207 28641 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:57:33.343279 28633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:33.344009 28633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:33.349472 28633 catalog_manager.cc:1347] Generated new cluster ID: ceeb4eed891d46d4a30570b31dad26f7
I20260809 07:57:33.349537 28633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:33.371492 28633 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:33.371820 28633 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:33.382007 28633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f2470b0ed3e4938a7aabb8169b240c5: Generated new TSK 0
I20260809 07:57:33.382706 28633 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:33.387336 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 28498
2026-08-09T07:57:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1483 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:33 dist-test-slave-vpvm krb5kdc[28651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:33 dist-test-slave-vpvm krb5kdc[28651](info): set up 2 sockets
Aug 09 07:57:33 dist-test-slave-vpvm krb5kdc[28651](info): commencing operation
krb5kdc: starting...
W20260809 07:57:34.516830 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:34 dist-test-slave-vpvm krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 07:57:34.638458 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:34.748445 28669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:34.748687 28669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:34.748732 28669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:34.752625 28669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:34.752679 28669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:34.752699 28669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:34.752729 28669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:34.752755 28669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:34.757773 28669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:34.758719 28669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:34.759658 28669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:34.767733 28678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:34.767702 28680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:34.767812 28676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:34.768059 28669 server_base.cc:1034] running on GCE node
I20260809 07:57:34.768597 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:34.769626 28669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:34.770782 28669 hybrid_clock.cc:648] HybridClock initialized: now 1786262254770764 us; error 30 us; skew 500 ppm
Aug 09 07:57:34 dist-test-slave-vpvm krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:34.775969 28669 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 07:57:34.776799 28669 webserver.cc:460] Webserver started at http://127.0.0.1:40619/ using document root <none> and password file <none>
I20260809 07:57:34.777318 28669 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:34.777375 28669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:34.777534 28669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:34.779462 28669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "4221346ce2d244ea99d7dafefeb36616"
format_stamp: "Formatted at 2026-08-09 07:57:34 on dist-test-slave-vpvm"
server_key: "e497f85d68b6ad751c562c6a48fb33e9"
server_key_iv: "7b07ffa88e7b231c3cc57d26cbdbcaac"
server_key_version: "encryptionkey@0"
I20260809 07:57:34.779958 28669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "4221346ce2d244ea99d7dafefeb36616"
format_stamp: "Formatted at 2026-08-09 07:57:34 on dist-test-slave-vpvm"
server_key: "e497f85d68b6ad751c562c6a48fb33e9"
server_key_iv: "7b07ffa88e7b231c3cc57d26cbdbcaac"
server_key_version: "encryptionkey@0"
I20260809 07:57:34.783416 28669 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 07:57:34.785660 28689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:34.786866 28669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:57:34.786984 28669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "4221346ce2d244ea99d7dafefeb36616"
format_stamp: "Formatted at 2026-08-09 07:57:34 on dist-test-slave-vpvm"
server_key: "e497f85d68b6ad751c562c6a48fb33e9"
server_key_iv: "7b07ffa88e7b231c3cc57d26cbdbcaac"
server_key_version: "encryptionkey@0"
I20260809 07:57:34.787078 28669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:34.797160 28669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:34.799650 28669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:34.799800 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:34.805985 28669 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39477
I20260809 07:57:34.806067 28765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39477 every 8 connection(s)
I20260809 07:57:34.806864 28669 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:34.809609 28766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:34.814838 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 28669
I20260809 07:57:34.814996 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:34.815289 28766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616: Bootstrap starting.
I20260809 07:57:34.815338 29120 external_mini_cluster.cc:1428] Setting key cebdd277429c875f367c064062d119c3
I20260809 07:57:34.817886 28766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:34.818674 28766 log.cc:824] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:34.820538 28766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616: No bootstrap required, opened a new log
Aug 09 07:57:34 dist-test-slave-vpvm krb5kdc[28651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 07:57:34.824419 28766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4221346ce2d244ea99d7dafefeb36616" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39477 } }
I20260809 07:57:34.824635 28766 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:34.824702 28766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4221346ce2d244ea99d7dafefeb36616, State: Initialized, Role: FOLLOWER
I20260809 07:57:34.825140 28766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4221346ce2d244ea99d7dafefeb36616" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39477 } }
I20260809 07:57:34.825281 28766 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:34.825356 28766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:34.825480 28766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:34.826387 28766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4221346ce2d244ea99d7dafefeb36616" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39477 } }
I20260809 07:57:34.826720 28766 leader_election.cc:302] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4221346ce2d244ea99d7dafefeb36616; no voters:
I20260809 07:57:34.827036 28766 leader_election.cc:288] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:34.827126 28772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:34.827316 28772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616 [term 1 LEADER]: Becoming Leader. State: Replica: 4221346ce2d244ea99d7dafefeb36616, State: Running, Role: LEADER
I20260809 07:57:34.827690 28772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4221346ce2d244ea99d7dafefeb36616" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39477 } }
I20260809 07:57:34.828357 28766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:34.829350 28776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4221346ce2d244ea99d7dafefeb36616. Latest consensus state: current_term: 1 leader_uuid: "4221346ce2d244ea99d7dafefeb36616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4221346ce2d244ea99d7dafefeb36616" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39477 } } }
I20260809 07:57:34.829493 28776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:34.829710 28769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:34.816670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:53909 (local address 127.0.0.1:39477)
0809 07:57:34.817109 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:34.817124 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:34.817160 (+ 36us) server_negotiation.cc:407] Connection header received
0809 07:57:34.817765 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:34.817782 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:34.818179 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:34.818479 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:34.819275 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:34.820249 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:34.821144 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:34.821602 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:34.821762 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:34.824499 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:34.824506 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:34.824521 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:34.824547 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:34.826252 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:34.826767 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:34.826772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:34.826777 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:34.826913 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:34.827296 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:34.827299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:34.827301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:34.827831 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:34.828041 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:34.828344 (+ 303us) server_negotiation.cc:299] Negotiation successful
0809 07:57:34.828734 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":151,"threads_started":1}
I20260809 07:57:34.829861 28774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4221346ce2d244ea99d7dafefeb36616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4221346ce2d244ea99d7dafefeb36616" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39477 } } }
I20260809 07:57:34.830029 28774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:34.830783 28785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:34.833762 28785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:34.834141 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:57:34.840844 28785 catalog_manager.cc:1347] Generated new cluster ID: 3da26f40db104925bc245eba4551b76b
I20260809 07:57:34.840920 28785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:34.844969 28769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:34.836428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:34497 (local address 127.0.0.1:39477)
0809 07:57:34.836548 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:34.836553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:34.836782 (+ 229us) server_negotiation.cc:407] Connection header received
0809 07:57:34.836905 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:34.836909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:34.836960 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:34.837031 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:34.837744 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:34.838442 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:34.839249 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:34.839600 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:34.839720 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:34.841347 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:34.841352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:34.841355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:34.841382 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:34.843073 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:34.843715 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:34.843719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:34.843721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:34.843807 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:34.844121 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:34.844124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:34.844126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:34.844328 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:34.844479 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:34.844710 (+ 231us) server_negotiation.cc:299] Negotiation successful
0809 07:57:34.844802 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 07:57:34.848205 28785 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:34.849350 28785 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:34.863312 28785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4221346ce2d244ea99d7dafefeb36616: Generated new TSK 0
I20260809 07:57:34.864094 28785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:34.873389 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 28669
W20260809 07:57:34.884944 28646 connection.cc:535] client connection to 127.0.0.1:39477 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39477): Connection reset by peer (error 104)
2026-08-09T07:57:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1476 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:34 dist-test-slave-vpvm krb5kdc[28810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:34 dist-test-slave-vpvm krb5kdc[28810](info): set up 2 sockets
Aug 09 07:57:34 dist-test-slave-vpvm krb5kdc[28810](info): commencing operation
krb5kdc: starting...
W20260809 07:57:36.039768 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:36 dist-test-slave-vpvm krb5kdc[28810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 07:57:36.165023 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40069
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:36.282804 28826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:36.283037 28826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:36.283078 28826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:36.286972 28826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:36.287024 28826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:36.287041 28826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:36.287060 28826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:36.287087 28826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:36.292137 28826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:36.293047 28826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:36.294023 28826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:36.301913 28834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:36.301944 28833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:36.301925 28836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:36.302172 28826 server_base.cc:1034] running on GCE node
I20260809 07:57:36.302764 28826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:36.303830 28826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:36.304994 28826 hybrid_clock.cc:648] HybridClock initialized: now 1786262256304972 us; error 31 us; skew 500 ppm
Aug 09 07:57:36 dist-test-slave-vpvm krb5kdc[28810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:36.307627 28826 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 07:57:36.308470 28826 webserver.cc:460] Webserver started at http://127.0.0.1:43035/ using document root <none> and password file <none>
I20260809 07:57:36.308959 28826 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:36.309015 28826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:36.309187 28826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:36.311167 28826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "f0d6df5d18be437eb6ce928a2ec079cf"
format_stamp: "Formatted at 2026-08-09 07:57:36 on dist-test-slave-vpvm"
server_key: "a8ab4da1c0c7a3d23c22920577a7cb28"
server_key_iv: "93ddbc82dcafbdb0c77bdd96e7a4d33e"
server_key_version: "encryptionkey@0"
I20260809 07:57:36.311685 28826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "f0d6df5d18be437eb6ce928a2ec079cf"
format_stamp: "Formatted at 2026-08-09 07:57:36 on dist-test-slave-vpvm"
server_key: "a8ab4da1c0c7a3d23c22920577a7cb28"
server_key_iv: "93ddbc82dcafbdb0c77bdd96e7a4d33e"
server_key_version: "encryptionkey@0"
I20260809 07:57:36.315223 28826 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 07:57:36.317497 28844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:36.318758 28826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:57:36.318886 28826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "f0d6df5d18be437eb6ce928a2ec079cf"
format_stamp: "Formatted at 2026-08-09 07:57:36 on dist-test-slave-vpvm"
server_key: "a8ab4da1c0c7a3d23c22920577a7cb28"
server_key_iv: "93ddbc82dcafbdb0c77bdd96e7a4d33e"
server_key_version: "encryptionkey@0"
I20260809 07:57:36.318977 28826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:36.334776 28826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:36.337760 28826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:36.337937 28826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:36.344712 28826 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40069
I20260809 07:57:36.344748 28919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40069 every 8 connection(s)
I20260809 07:57:36.345680 28826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:36.348449 28921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:36.351104 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 28826
I20260809 07:57:36.351235 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:36.351576 29120 external_mini_cluster.cc:1428] Setting key 8281678beaed89f81608b82f5d8de102
I20260809 07:57:36.354074 28921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf: Bootstrap starting.
I20260809 07:57:36.356372 28921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:36.357244 28921 log.cc:824] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:36.359030 28921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf: No bootstrap required, opened a new log
I20260809 07:57:36.361094 28925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:36.352928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:39329 (local address 127.0.0.1:40069)
0809 07:57:36.353341 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:36.353359 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:36.353398 (+ 39us) server_negotiation.cc:407] Connection header received
0809 07:57:36.354073 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:36.354094 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:36.354460 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:36.354736 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:36.355072 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:36.356048 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:36.357259 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:36.357777 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:36.358001 (+ 224us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:36.358882 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:36.358888 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:36.358913 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:36.359340 (+ 427us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:36.359409 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:36.359436 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:36.359567 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:36.359837 (+ 270us) server_negotiation.cc:299] Negotiation successful
0809 07:57:36.359966 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":122,"threads_started":1}
I20260809 07:57:36.362716 28921 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d6df5d18be437eb6ce928a2ec079cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40069 } }
I20260809 07:57:36.362970 28921 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:36.363039 28921 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d6df5d18be437eb6ce928a2ec079cf, State: Initialized, Role: FOLLOWER
I20260809 07:57:36.363497 28921 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d6df5d18be437eb6ce928a2ec079cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40069 } }
I20260809 07:57:36.363658 28921 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:36.363723 28921 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:36.363812 28921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:36.364411 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:57:36.364797 28921 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d6df5d18be437eb6ce928a2ec079cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40069 } }
I20260809 07:57:36.365166 28921 leader_election.cc:302] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0d6df5d18be437eb6ce928a2ec079cf; no voters:
I20260809 07:57:36.365502 28921 leader_election.cc:288] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:36.365593 28928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:36.365803 28928 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf [term 1 LEADER]: Becoming Leader. State: Replica: f0d6df5d18be437eb6ce928a2ec079cf, State: Running, Role: LEADER
I20260809 07:57:36.366191 28928 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d6df5d18be437eb6ce928a2ec079cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40069 } }
I20260809 07:57:36.366895 28921 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:36.367888 28933 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0d6df5d18be437eb6ce928a2ec079cf. Latest consensus state: current_term: 1 leader_uuid: "f0d6df5d18be437eb6ce928a2ec079cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d6df5d18be437eb6ce928a2ec079cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40069 } } }
I20260809 07:57:36.368018 28933 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:36.368389 28932 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0d6df5d18be437eb6ce928a2ec079cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d6df5d18be437eb6ce928a2ec079cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40069 } } }
I20260809 07:57:36.368482 28932 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:36.369240 28948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:36.371515 28925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:36.366747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:59489 (local address 127.0.0.1:40069)
0809 07:57:36.366847 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:36.366851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:36.367154 (+ 303us) server_negotiation.cc:407] Connection header received
0809 07:57:36.367265 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:36.367267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:36.367316 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:36.367391 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:36.367788 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:36.368734 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:36.369832 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:36.370079 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:36.370157 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:36.371036 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:36.371040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:36.371043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:36.371057 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:36.371084 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:36.371098 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:36.371181 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:36.371351 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 07:57:36.371384 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 07:57:36.372442 28948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:36.377765 28948 catalog_manager.cc:1347] Generated new cluster ID: aaad1d94b5f64a00ad107c1ef94e29b9
I20260809 07:57:36.377828 28948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:36.387328 28948 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:36.388159 28948 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:36.399616 28948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0d6df5d18be437eb6ce928a2ec079cf: Generated new TSK 0
I20260809 07:57:36.400355 28948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:36.405786 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 28826
W20260809 07:57:36.421813 28808 connection.cc:535] client connection to 127.0.0.1:40069 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40069): Connection reset by peer (error 104)
2026-08-09T07:57:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1537 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:36 dist-test-slave-vpvm krb5kdc[28963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:36 dist-test-slave-vpvm krb5kdc[28963](info): set up 2 sockets
Aug 09 07:57:36 dist-test-slave-vpvm krb5kdc[28963](info): commencing operation
krb5kdc: starting...
W20260809 07:57:37.562713 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:37 dist-test-slave-vpvm krb5kdc[28963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 07:57:37.678145 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:37.785382 28983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:37.785622 28983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:37.785662 28983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:37.789510 28983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:37.789564 28983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:37.789582 28983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:37.789602 28983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:37.789616 28983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:37.794584 28983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:37.795465 28983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:37.796360 28983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:37.804184 28993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:37.804177 28988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:37.804183 28989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:37.804370 28983 server_base.cc:1034] running on GCE node
I20260809 07:57:37.804921 28983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:37.806001 28983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:37.807170 28983 hybrid_clock.cc:648] HybridClock initialized: now 1786262257807160 us; error 28 us; skew 500 ppm
Aug 09 07:57:37 dist-test-slave-vpvm krb5kdc[28963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:37.809684 28983 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 07:57:37.810549 28983 webserver.cc:460] Webserver started at http://127.0.0.1:43959/ using document root <none> and password file <none>
I20260809 07:57:37.811053 28983 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:37.811117 28983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:37.811332 28983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:37.813280 28983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "ac0b0190aafc419e89d94ea2a1405a25"
format_stamp: "Formatted at 2026-08-09 07:57:37 on dist-test-slave-vpvm"
server_key: "6b8d6d652be735e37fe184e9c7fd85ba"
server_key_iv: "ffee030129a4cc9aae64fc33e91ba830"
server_key_version: "encryptionkey@0"
I20260809 07:57:37.813796 28983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "ac0b0190aafc419e89d94ea2a1405a25"
format_stamp: "Formatted at 2026-08-09 07:57:37 on dist-test-slave-vpvm"
server_key: "6b8d6d652be735e37fe184e9c7fd85ba"
server_key_iv: "ffee030129a4cc9aae64fc33e91ba830"
server_key_version: "encryptionkey@0"
I20260809 07:57:37.817255 28983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 07:57:37.819613 29000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:37.820746 28983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:57:37.820865 28983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "ac0b0190aafc419e89d94ea2a1405a25"
format_stamp: "Formatted at 2026-08-09 07:57:37 on dist-test-slave-vpvm"
server_key: "6b8d6d652be735e37fe184e9c7fd85ba"
server_key_iv: "ffee030129a4cc9aae64fc33e91ba830"
server_key_version: "encryptionkey@0"
I20260809 07:57:37.820958 28983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:37.825487 28983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:37.828537 28983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:37.828712 28983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:37.835227 28983 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37169
I20260809 07:57:37.835289 29070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37169 every 8 connection(s)
I20260809 07:57:37.836144 28983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:37.838973 29072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:37.842969 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 28983
I20260809 07:57:37.843078 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:37.843354 29120 external_mini_cluster.cc:1428] Setting key 41a7474f01cd1fc955cbaec3edd7af90
I20260809 07:57:37.844610 29072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25: Bootstrap starting.
I20260809 07:57:37.846868 29072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:37.847738 29072 log.cc:824] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:37.848588 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:37.844772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:38573 (local address 127.0.0.1:37169)
0809 07:57:37.845186 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:37.845201 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:37.845236 (+ 35us) server_negotiation.cc:407] Connection header received
0809 07:57:37.845839 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:37.845856 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:37.846186 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:37.846457 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:37.847293 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:37.847301 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:37.847318 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:37.847332 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:37.847401 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:37.847425 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:37.847518 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:37.847636 (+ 118us) server_negotiation.cc:299] Negotiation successful
0809 07:57:37.847781 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":119,"threads_started":1}
I20260809 07:57:37.849710 29072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25: No bootstrap required, opened a new log
I20260809 07:57:37.851372 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:57:37.853489 29072 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac0b0190aafc419e89d94ea2a1405a25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37169 } }
I20260809 07:57:37.853724 29072 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:37.853780 29072 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac0b0190aafc419e89d94ea2a1405a25, State: Initialized, Role: FOLLOWER
I20260809 07:57:37.854230 29072 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac0b0190aafc419e89d94ea2a1405a25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37169 } }
I20260809 07:57:37.854364 29072 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:37.854430 29072 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:37.854524 29072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:37.854910 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:37.853270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:49453 (local address 127.0.0.1:37169)
0809 07:57:37.853388 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:37.853392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:37.853599 (+ 207us) server_negotiation.cc:407] Connection header received
0809 07:57:37.853700 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:37.853703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:37.853734 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:37.853802 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:37.854565 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:37.854568 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:37.854569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:37.854580 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:37.854605 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:37.854617 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:37.854693 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:37.854762 (+ 69us) server_negotiation.cc:299] Negotiation successful
0809 07:57:37.854803 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 07:57:37.855525 29072 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac0b0190aafc419e89d94ea2a1405a25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37169 } }
I20260809 07:57:37.855866 29072 leader_election.cc:302] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac0b0190aafc419e89d94ea2a1405a25; no voters:
I20260809 07:57:37.856178 29072 leader_election.cc:288] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:37.856225 29088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:37.856491 29088 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25 [term 1 LEADER]: Becoming Leader. State: Replica: ac0b0190aafc419e89d94ea2a1405a25, State: Running, Role: LEADER
I20260809 07:57:37.856813 29088 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac0b0190aafc419e89d94ea2a1405a25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37169 } }
I20260809 07:57:37.857301 29072 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:37.858400 29092 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac0b0190aafc419e89d94ea2a1405a25. Latest consensus state: current_term: 1 leader_uuid: "ac0b0190aafc419e89d94ea2a1405a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac0b0190aafc419e89d94ea2a1405a25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37169 } } }
I20260809 07:57:37.858526 29092 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:37.858820 29091 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac0b0190aafc419e89d94ea2a1405a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac0b0190aafc419e89d94ea2a1405a25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37169 } } }
I20260809 07:57:37.858907 29091 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:37.858947 29102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:37.862027 29102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:37.867475 29102 catalog_manager.cc:1347] Generated new cluster ID: 8bc70f674e4f4c7da95e8ba9cd248602
I20260809 07:57:37.867524 29102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:37.896059 29102 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:37.896422 29102 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:37.902309 29102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac0b0190aafc419e89d94ea2a1405a25: Generated new TSK 0
I20260809 07:57:37.902927 29102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:37.913718 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 28983
2026-08-09T07:57:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1503 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:37 dist-test-slave-vpvm krb5kdc[29116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:37 dist-test-slave-vpvm krb5kdc[29116](info): set up 2 sockets
Aug 09 07:57:37 dist-test-slave-vpvm krb5kdc[29116](info): commencing operation
krb5kdc: starting...
W20260809 07:57:39.042670 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:39 dist-test-slave-vpvm krb5kdc[29116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 07:57:39.158468 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:39.268743 29147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:39.268986 29147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:39.269052 29147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:39.272921 29147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:39.272974 29147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:39.273008 29147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:39.273033 29147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:39.273061 29147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:39.278059 29147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:39.279038 29147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:39.279968 29147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:39.287869 29156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:39.287882 29157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:39.287918 29159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:39.288187 29147 server_base.cc:1034] running on GCE node
I20260809 07:57:39.288698 29147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:39.289738 29147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:39.290911 29147 hybrid_clock.cc:648] HybridClock initialized: now 1786262259290885 us; error 36 us; skew 500 ppm
Aug 09 07:57:39 dist-test-slave-vpvm krb5kdc[29116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:39.293459 29147 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 07:57:39.294287 29147 webserver.cc:460] Webserver started at http://127.0.0.1:42299/ using document root <none> and password file <none>
I20260809 07:57:39.294780 29147 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:39.294835 29147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:39.294997 29147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:39.296861 29147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "f1ef9edf6294406e8e472656ce9be7b8"
format_stamp: "Formatted at 2026-08-09 07:57:39 on dist-test-slave-vpvm"
server_key: "cdc51fbe374f1d3a28d00b24d4c4b885"
server_key_iv: "2bece5facab5352d4acd97176a9b4aab"
server_key_version: "encryptionkey@0"
I20260809 07:57:39.297356 29147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "f1ef9edf6294406e8e472656ce9be7b8"
format_stamp: "Formatted at 2026-08-09 07:57:39 on dist-test-slave-vpvm"
server_key: "cdc51fbe374f1d3a28d00b24d4c4b885"
server_key_iv: "2bece5facab5352d4acd97176a9b4aab"
server_key_version: "encryptionkey@0"
I20260809 07:57:39.300769 29147 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:57:39.303032 29178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:39.304175 29147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:57:39.304275 29147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "f1ef9edf6294406e8e472656ce9be7b8"
format_stamp: "Formatted at 2026-08-09 07:57:39 on dist-test-slave-vpvm"
server_key: "cdc51fbe374f1d3a28d00b24d4c4b885"
server_key_iv: "2bece5facab5352d4acd97176a9b4aab"
server_key_version: "encryptionkey@0"
I20260809 07:57:39.304353 29147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:39.330578 29147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:39.333387 29147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:39.333562 29147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:39.340814 29147 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44135
I20260809 07:57:39.340858 29258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44135 every 8 connection(s)
I20260809 07:57:39.341777 29147 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:39.344681 29259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:39.344918 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 29147
I20260809 07:57:39.345046 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:39.345335 29120 external_mini_cluster.cc:1428] Setting key e7ef35941d65371002fa210efeee92af
I20260809 07:57:39.351907 29259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8: Bootstrap starting.
I20260809 07:57:39.354375 29259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8: Neither blocks nor log segments found. Creating new log.
Aug 09 07:57:39 dist-test-slave-vpvm krb5kdc[29116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 07:57:39.355273 29259 log.cc:824] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:39.357312 29259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8: No bootstrap required, opened a new log
I20260809 07:57:39.359694 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:39.346777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:60813 (local address 127.0.0.1:44135)
0809 07:57:39.347225 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:39.347245 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:39.347292 (+ 47us) server_negotiation.cc:407] Connection header received
0809 07:57:39.348032 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:39.348058 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:39.348519 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:39.348896 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:39.349551 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:39.351053 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:39.351873 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:39.352536 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:39.352688 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:39.355188 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:39.355193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:39.355207 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:39.355234 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:39.356875 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:39.357433 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:39.357439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:39.357445 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:39.357554 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:39.357880 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:39.357882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:39.357883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:39.358310 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:39.358464 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:39.358761 (+ 297us) server_negotiation.cc:299] Negotiation successful
0809 07:57:39.358949 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":127,"threads_started":1}
I20260809 07:57:39.361168 29259 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ef9edf6294406e8e472656ce9be7b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } }
I20260809 07:57:39.361444 29259 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:39.361531 29259 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ef9edf6294406e8e472656ce9be7b8, State: Initialized, Role: FOLLOWER
I20260809 07:57:39.362033 29259 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ef9edf6294406e8e472656ce9be7b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } }
I20260809 07:57:39.362181 29259 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:39.362250 29259 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:39.362345 29259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:39.362666 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:57:39.363404 29259 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ef9edf6294406e8e472656ce9be7b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } }
I20260809 07:57:39.363781 29259 leader_election.cc:302] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1ef9edf6294406e8e472656ce9be7b8; no voters:
I20260809 07:57:39.364120 29259 leader_election.cc:288] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:39.364225 29270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:39.364483 29270 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8 [term 1 LEADER]: Becoming Leader. State: Replica: f1ef9edf6294406e8e472656ce9be7b8, State: Running, Role: LEADER
I20260809 07:57:39.364791 29270 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ef9edf6294406e8e472656ce9be7b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } }
I20260809 07:57:39.365365 29259 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:39.366075 29270 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1ef9edf6294406e8e472656ce9be7b8. Latest consensus state: current_term: 1 leader_uuid: "f1ef9edf6294406e8e472656ce9be7b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ef9edf6294406e8e472656ce9be7b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } } }
I20260809 07:57:39.366212 29270 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:39.366864 29273 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1ef9edf6294406e8e472656ce9be7b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ef9edf6294406e8e472656ce9be7b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } } }
I20260809 07:57:39.367030 29273 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8 [sys.catalog]: This master's current role is: LEADER
W20260809 07:57:39.370517 29294 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:57:39.370651 29294 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:57:39.370785 29284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:39.371764 29284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:39.374986 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:39.364911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:49175 (local address 127.0.0.1:44135)
0809 07:57:39.365009 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:39.365013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:39.366304 (+ 1291us) server_negotiation.cc:407] Connection header received
0809 07:57:39.366507 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:39.366510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:39.366558 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:39.366607 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:39.367528 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:39.368223 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:39.369228 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:39.369547 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:39.369649 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:39.371386 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:39.371390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:39.371393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:39.371418 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:39.372996 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:39.373594 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:39.373598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:39.373600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:39.373700 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:39.374081 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:39.374085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:39.374087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:39.374297 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:39.374459 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:39.374735 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 07:57:39.374834 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260809 07:57:39.377737 29284 catalog_manager.cc:1347] Generated new cluster ID: 2fe182649cc84173bb3b8b5393846c1c
I20260809 07:57:39.377789 29284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:39.385886 29284 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:39.386749 29284 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:39.395893 29284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1ef9edf6294406e8e472656ce9be7b8: Generated new TSK 0
I20260809 07:57:39.396494 29284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:39.402271 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 29147
W20260809 07:57:39.418659 29114 connection.cc:535] client connection to 127.0.0.1:44135 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44135): Connection reset by peer (error 104)
2026-08-09T07:57:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1493 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:39 dist-test-slave-vpvm krb5kdc[29302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:39 dist-test-slave-vpvm krb5kdc[29302](info): set up 2 sockets
Aug 09 07:57:39 dist-test-slave-vpvm krb5kdc[29302](info): commencing operation
krb5kdc: starting...
W20260809 07:57:40.536736 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:40 dist-test-slave-vpvm krb5kdc[29302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 07:57:40.652699 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41321
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:40.767120 29324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:40.767372 29324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:40.767458 29324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:40.771418 29324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:40.771474 29324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:40.771495 29324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:40.771530 29324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:40.771562 29324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:40.776707 29324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41321
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:40.777685 29324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:40.778703 29324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:40.786809 29337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:40.786805 29333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:40.786820 29334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:40.786918 29324 server_base.cc:1034] running on GCE node
I20260809 07:57:40.787650 29324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:40.788728 29324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:40.789885 29324 hybrid_clock.cc:648] HybridClock initialized: now 1786262260789866 us; error 32 us; skew 500 ppm
Aug 09 07:57:40 dist-test-slave-vpvm krb5kdc[29302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:40.795476 29324 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 07:57:40.796342 29324 webserver.cc:460] Webserver started at http://127.0.0.1:38699/ using document root <none> and password file <none>
I20260809 07:57:40.796849 29324 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:40.796911 29324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:40.797089 29324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:40.799078 29324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "bf59acd225d74a43adeace68a443c76c"
format_stamp: "Formatted at 2026-08-09 07:57:40 on dist-test-slave-vpvm"
server_key: "c50a5da625386eef20f1d4434d8b8e79"
server_key_iv: "9c7d9241644875521256c282fdb65df0"
server_key_version: "encryptionkey@0"
I20260809 07:57:40.799608 29324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "bf59acd225d74a43adeace68a443c76c"
format_stamp: "Formatted at 2026-08-09 07:57:40 on dist-test-slave-vpvm"
server_key: "c50a5da625386eef20f1d4434d8b8e79"
server_key_iv: "9c7d9241644875521256c282fdb65df0"
server_key_version: "encryptionkey@0"
I20260809 07:57:40.803184 29324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:57:40.805544 29347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:40.806742 29324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:57:40.806855 29324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "bf59acd225d74a43adeace68a443c76c"
format_stamp: "Formatted at 2026-08-09 07:57:40 on dist-test-slave-vpvm"
server_key: "c50a5da625386eef20f1d4434d8b8e79"
server_key_iv: "9c7d9241644875521256c282fdb65df0"
server_key_version: "encryptionkey@0"
I20260809 07:57:40.806944 29324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:40.828015 29324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:40.830762 29324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:40.830935 29324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:40.837692 29324 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41321
I20260809 07:57:40.837704 29419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41321 every 8 connection(s)
I20260809 07:57:40.838680 29324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:40.841574 29420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:40.847114 29420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c: Bootstrap starting.
I20260809 07:57:40.848295 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 29324
I20260809 07:57:40.848423 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:40.848683 29120 external_mini_cluster.cc:1428] Setting key ef20778c0f1244c50adbfe6967a1a453
I20260809 07:57:40.849633 29420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:40.850486 29420 log.cc:824] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:40.852296 29420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c: No bootstrap required, opened a new log
I20260809 07:57:40.855846 29420 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf59acd225d74a43adeace68a443c76c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41321 } }
I20260809 07:57:40.856069 29420 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:40.856133 29420 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf59acd225d74a43adeace68a443c76c, State: Initialized, Role: FOLLOWER
I20260809 07:57:40.856606 29420 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf59acd225d74a43adeace68a443c76c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41321 } }
I20260809 07:57:40.856765 29420 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:40.856840 29420 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:40.856943 29420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:40.857396 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:40.849794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:45971 (local address 127.0.0.1:41321)
0809 07:57:40.850289 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:40.850305 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:40.850340 (+ 35us) server_negotiation.cc:407] Connection header received
0809 07:57:40.850983 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:40.851002 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:40.851345 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:40.851646 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:40.852046 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:40.853122 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:40.854007 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:40.854484 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:40.854641 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:40.855487 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:40.855492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:40.855504 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:40.855879 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:40.855939 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:40.855963 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:40.856076 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:40.856303 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 07:57:40.856444 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":175,"threads_started":1}
I20260809 07:57:40.857923 29420 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf59acd225d74a43adeace68a443c76c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41321 } }
I20260809 07:57:40.858273 29420 leader_election.cc:302] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf59acd225d74a43adeace68a443c76c; no voters:
I20260809 07:57:40.858578 29420 leader_election.cc:288] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:40.858659 29426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:40.858886 29426 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c [term 1 LEADER]: Becoming Leader. State: Replica: bf59acd225d74a43adeace68a443c76c, State: Running, Role: LEADER
I20260809 07:57:40.859254 29426 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf59acd225d74a43adeace68a443c76c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41321 } }
I20260809 07:57:40.859797 29420 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:40.860432 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:57:40.860908 29427 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf59acd225d74a43adeace68a443c76c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf59acd225d74a43adeace68a443c76c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41321 } } }
I20260809 07:57:40.861023 29427 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:40.861310 29428 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf59acd225d74a43adeace68a443c76c. Latest consensus state: current_term: 1 leader_uuid: "bf59acd225d74a43adeace68a443c76c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf59acd225d74a43adeace68a443c76c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41321 } } }
I20260809 07:57:40.861388 29428 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c [sys.catalog]: This master's current role is: LEADER
W20260809 07:57:40.864635 29453 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:57:40.864699 29453 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:57:40.864766 29442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:40.865593 29442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:40.867010 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:40.862735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:42959 (local address 127.0.0.1:41321)
0809 07:57:40.862843 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:40.862847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:40.863072 (+ 225us) server_negotiation.cc:407] Connection header received
0809 07:57:40.863194 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:40.863196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:40.863243 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:40.863323 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:40.863599 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:40.864302 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:40.865355 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:40.865605 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:40.865692 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:40.866554 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:40.866558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:40.866561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:40.866574 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:40.866602 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:40.866615 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:40.866702 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:40.866851 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 07:57:40.866886 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 07:57:40.870833 29442 catalog_manager.cc:1347] Generated new cluster ID: fa31b174beff46f388e7c3c88db9b0c1
I20260809 07:57:40.870899 29442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:40.892856 29442 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:40.894173 29442 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:40.901800 29442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf59acd225d74a43adeace68a443c76c: Generated new TSK 0
I20260809 07:57:40.902604 29442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:40.913733 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 29324
W20260809 07:57:40.929688 29300 connection.cc:535] client connection to 127.0.0.1:41321 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41321): Connection reset by peer (error 104)
2026-08-09T07:57:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1511 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:40 dist-test-slave-vpvm krb5kdc[29466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:40 dist-test-slave-vpvm krb5kdc[29466](info): set up 2 sockets
Aug 09 07:57:40 dist-test-slave-vpvm krb5kdc[29466](info): commencing operation
krb5kdc: starting...
W20260809 07:57:42.059669 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:42 dist-test-slave-vpvm krb5kdc[29466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 07:57:42.178030 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45339
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:42.289623 29485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:42.289868 29485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:42.289907 29485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:42.293763 29485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:42.293814 29485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:42.293833 29485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:42.293846 29485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:42.293866 29485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:42.298895 29485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45339
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:42.299779 29485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:42.300683 29485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:42.308555 29491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:42.308568 29492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:42.308605 29494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:42.308772 29485 server_base.cc:1034] running on GCE node
I20260809 07:57:42.309396 29485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:42.310494 29485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:42.311657 29485 hybrid_clock.cc:648] HybridClock initialized: now 1786262262311643 us; error 25 us; skew 500 ppm
Aug 09 07:57:42 dist-test-slave-vpvm krb5kdc[29466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:42.314306 29485 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 07:57:42.315158 29485 webserver.cc:460] Webserver started at http://127.0.0.1:40417/ using document root <none> and password file <none>
I20260809 07:57:42.315665 29485 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:42.315729 29485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:42.315927 29485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:42.317898 29485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "2a73c027383843bebaf6e80a2892091a"
format_stamp: "Formatted at 2026-08-09 07:57:42 on dist-test-slave-vpvm"
server_key: "2319ee030f3ac6ba5fe50d055392dca3"
server_key_iv: "345df2f787ba2f39c90705a5de11ca43"
server_key_version: "encryptionkey@0"
I20260809 07:57:42.318497 29485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "2a73c027383843bebaf6e80a2892091a"
format_stamp: "Formatted at 2026-08-09 07:57:42 on dist-test-slave-vpvm"
server_key: "2319ee030f3ac6ba5fe50d055392dca3"
server_key_iv: "345df2f787ba2f39c90705a5de11ca43"
server_key_version: "encryptionkey@0"
I20260809 07:57:42.322036 29485 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 07:57:42.324322 29512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:42.325445 29485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:57:42.325560 29485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "2a73c027383843bebaf6e80a2892091a"
format_stamp: "Formatted at 2026-08-09 07:57:42 on dist-test-slave-vpvm"
server_key: "2319ee030f3ac6ba5fe50d055392dca3"
server_key_iv: "345df2f787ba2f39c90705a5de11ca43"
server_key_version: "encryptionkey@0"
I20260809 07:57:42.325652 29485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:42.330119 29485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:42.333151 29485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:42.333320 29485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:42.340180 29485 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45339
I20260809 07:57:42.340185 29588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45339 every 8 connection(s)
I20260809 07:57:42.341085 29485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:42.343523 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 29485
I20260809 07:57:42.343650 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:42.343914 29120 external_mini_cluster.cc:1428] Setting key 0933c4292510ec9075cf272f79b8f689
I20260809 07:57:42.344101 29589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:42.348964 29593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:42.345204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:58227 (local address 127.0.0.1:45339)
0809 07:57:42.345622 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:42.345645 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:42.345681 (+ 36us) server_negotiation.cc:407] Connection header received
0809 07:57:42.346355 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:42.346380 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:42.346662 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:42.346933 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:42.347772 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:42.347776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:42.347788 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:42.347799 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:42.347852 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:42.347871 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:42.347978 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:42.348102 (+ 124us) server_negotiation.cc:299] Negotiation successful
0809 07:57:42.348249 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":117,"threads_started":1}
I20260809 07:57:42.350881 29589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a: Bootstrap starting.
I20260809 07:57:42.351667 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:57:42.353587 29589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:42.354529 29589 log.cc:824] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:42.355616 29593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:42.354062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:58839 (local address 127.0.0.1:45339)
0809 07:57:42.354181 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:42.354186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:42.354290 (+ 104us) server_negotiation.cc:407] Connection header received
0809 07:57:42.354401 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:42.354405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:42.354445 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:42.354541 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:42.355148 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:42.355152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:42.355155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:42.355170 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:42.355201 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:42.355216 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:42.355314 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:42.355446 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 07:57:42.355483 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 07:57:42.356801 29589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a: No bootstrap required, opened a new log
I20260809 07:57:42.360481 29589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a73c027383843bebaf6e80a2892091a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45339 } }
I20260809 07:57:42.360780 29589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:42.360845 29589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a73c027383843bebaf6e80a2892091a, State: Initialized, Role: FOLLOWER
I20260809 07:57:42.361276 29589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a73c027383843bebaf6e80a2892091a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45339 } }
I20260809 07:57:42.361411 29589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:42.361475 29589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:42.361573 29589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:42.362531 29589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a73c027383843bebaf6e80a2892091a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45339 } }
I20260809 07:57:42.362872 29589 leader_election.cc:302] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a73c027383843bebaf6e80a2892091a; no voters:
I20260809 07:57:42.363205 29589 leader_election.cc:288] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:42.363308 29603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:42.363519 29603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a [term 1 LEADER]: Becoming Leader. State: Replica: 2a73c027383843bebaf6e80a2892091a, State: Running, Role: LEADER
I20260809 07:57:42.363855 29603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a73c027383843bebaf6e80a2892091a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45339 } }
I20260809 07:57:42.364567 29589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:42.365533 29605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a73c027383843bebaf6e80a2892091a. Latest consensus state: current_term: 1 leader_uuid: "2a73c027383843bebaf6e80a2892091a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a73c027383843bebaf6e80a2892091a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45339 } } }
I20260809 07:57:42.365677 29605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:42.366015 29604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a73c027383843bebaf6e80a2892091a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a73c027383843bebaf6e80a2892091a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45339 } } }
I20260809 07:57:42.366099 29617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:42.366179 29604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:42.368853 29617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:42.374176 29617 catalog_manager.cc:1347] Generated new cluster ID: 9850697ec9a14fcf993553b76418e2b9
I20260809 07:57:42.374233 29617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:42.409539 29617 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:42.409874 29617 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:42.413700 29617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a73c027383843bebaf6e80a2892091a: Generated new TSK 0
I20260809 07:57:42.414342 29617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:42.420002 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 29485
2026-08-09T07:57:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1507 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:42 dist-test-slave-vpvm krb5kdc[29633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:42 dist-test-slave-vpvm krb5kdc[29633](info): set up 2 sockets
Aug 09 07:57:42 dist-test-slave-vpvm krb5kdc[29633](info): commencing operation
krb5kdc: starting...
W20260809 07:57:43.570385 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:43 dist-test-slave-vpvm krb5kdc[29633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:57:43.687011 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:38595
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:38595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:43.796533 29650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:43.796783 29650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:43.796841 29650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:43.800768 29650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:43.800822 29650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:43.800848 29650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:43.800870 29650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:43.800899 29650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:43.805902 29650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:38595
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:38595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:43.806938 29650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:43.807843 29650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:43.815719 29658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:43.815754 29659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:43.815840 29650 server_base.cc:1034] running on GCE node
W20260809 07:57:43.815753 29664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:43.816478 29650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:43.817500 29650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:43.818663 29650 hybrid_clock.cc:648] HybridClock initialized: now 1786262263818645 us; error 30 us; skew 500 ppm
Aug 09 07:57:43 dist-test-slave-vpvm krb5kdc[29633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262263, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:43.821224 29650 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:57:43.822117 29650 webserver.cc:460] Webserver started at http://127.28.112.62:41021/ using document root <none> and password file <none>
I20260809 07:57:43.822634 29650 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:43.822698 29650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:43.822896 29650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:43.824836 29650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "cbce3a0bca694dea8b0ab2df24b9b624"
format_stamp: "Formatted at 2026-08-09 07:57:43 on dist-test-slave-vpvm"
server_key: "cc8040cb2db6134411e23b558188092c"
server_key_iv: "404d56e3859918bef0a19d6fe4558e84"
server_key_version: "encryptionkey@0"
I20260809 07:57:43.825361 29650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "cbce3a0bca694dea8b0ab2df24b9b624"
format_stamp: "Formatted at 2026-08-09 07:57:43 on dist-test-slave-vpvm"
server_key: "cc8040cb2db6134411e23b558188092c"
server_key_iv: "404d56e3859918bef0a19d6fe4558e84"
server_key_version: "encryptionkey@0"
I20260809 07:57:43.828896 29650 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:57:43.831144 29672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:43.832276 29650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:57:43.832389 29650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "cbce3a0bca694dea8b0ab2df24b9b624"
format_stamp: "Formatted at 2026-08-09 07:57:43 on dist-test-slave-vpvm"
server_key: "cc8040cb2db6134411e23b558188092c"
server_key_iv: "404d56e3859918bef0a19d6fe4558e84"
server_key_version: "encryptionkey@0"
I20260809 07:57:43.832477 29650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:43.855827 29650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:43.859040 29650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:43.859222 29650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:43.866039 29650 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:38595
I20260809 07:57:43.866046 29743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:38595 every 8 connection(s)
I20260809 07:57:43.866976 29650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:43.869679 29744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:43.872601 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 29650
I20260809 07:57:43.872727 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:43.872994 29120 external_mini_cluster.cc:1428] Setting key e6aa6ae1079c396e3bc8117faba22306
I20260809 07:57:43.875496 29744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624: Bootstrap starting.
I20260809 07:57:43.878135 29744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:43.878928 29744 log.cc:824] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:43.880868 29744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624: No bootstrap required, opened a new log
Aug 09 07:57:43 dist-test-slave-vpvm krb5kdc[29633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:57:43.884954 29744 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbce3a0bca694dea8b0ab2df24b9b624" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38595 } }
I20260809 07:57:43.885231 29744 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:43.885296 29744 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbce3a0bca694dea8b0ab2df24b9b624, State: Initialized, Role: FOLLOWER
I20260809 07:57:43.885766 29744 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbce3a0bca694dea8b0ab2df24b9b624" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38595 } }
I20260809 07:57:43.885926 29744 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:43.886024 29744 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:43.886138 29744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:43.887185 29744 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbce3a0bca694dea8b0ab2df24b9b624" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38595 } }
I20260809 07:57:43.887529 29744 leader_election.cc:302] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbce3a0bca694dea8b0ab2df24b9b624; no voters:
I20260809 07:57:43.887574 29749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:43.874359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33675 (local address 127.28.112.62:38595)
0809 07:57:43.874778 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:43.874801 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:43.874844 (+ 43us) server_negotiation.cc:407] Connection header received
0809 07:57:43.875630 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:43.875654 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:43.876058 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:43.876400 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:43.877036 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:43.878487 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:43.879383 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:43.879956 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:43.880102 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:43.882654 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:43.882660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:43.882677 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:43.882709 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:43.884494 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:43.884966 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:43.884971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:43.884977 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:43.885108 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:43.885417 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:43.885420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:43.885422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:43.885900 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:43.886147 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:43.886441 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 07:57:43.886666 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":105,"threads_started":1}
I20260809 07:57:43.887893 29744 leader_election.cc:288] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:43.887997 29751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:43.888222 29751 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624 [term 1 LEADER]: Becoming Leader. State: Replica: cbce3a0bca694dea8b0ab2df24b9b624, State: Running, Role: LEADER
I20260809 07:57:43.889083 29744 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:43.888559 29751 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbce3a0bca694dea8b0ab2df24b9b624" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38595 } }
I20260809 07:57:43.890450 29751 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbce3a0bca694dea8b0ab2df24b9b624. Latest consensus state: current_term: 1 leader_uuid: "cbce3a0bca694dea8b0ab2df24b9b624" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbce3a0bca694dea8b0ab2df24b9b624" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38595 } } }
I20260809 07:57:43.890748 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:57:43.891048 29753 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbce3a0bca694dea8b0ab2df24b9b624" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbce3a0bca694dea8b0ab2df24b9b624" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 38595 } } }
I20260809 07:57:43.891201 29751 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:43.891606 29753 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624 [sys.catalog]: This master's current role is: LEADER
W20260809 07:57:43.894202 29775 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:57:43.894279 29775 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:57:43.894369 29776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:43.895040 29776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:43.900672 29776 catalog_manager.cc:1347] Generated new cluster ID: 400f9192fa3645a0b9ecb860c4dcac1e
I20260809 07:57:43.900729 29776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:43.901834 29749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:43.892882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44535 (local address 127.28.112.62:38595)
0809 07:57:43.892977 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:43.892980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:43.893202 (+ 222us) server_negotiation.cc:407] Connection header received
0809 07:57:43.893367 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:43.893370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:43.893414 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:43.893460 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:43.894416 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:43.895138 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:43.896195 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:43.896575 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:43.896683 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:43.898243 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:43.898246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:43.898249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:43.898274 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:43.899955 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:43.900431 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:43.900434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:43.900436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:43.900522 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:43.900928 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:43.900931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:43.900933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:43.901153 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:43.901301 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:43.901587 (+ 286us) server_negotiation.cc:299] Negotiation successful
0809 07:57:43.901687 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 07:57:43.911968 29776 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:43.912778 29776 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:43.921536 29776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbce3a0bca694dea8b0ab2df24b9b624: Generated new TSK 0
I20260809 07:57:43.922281 29776 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:43.929612 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 29650
W20260809 07:57:43.948787 29629 connection.cc:535] client connection to 127.28.112.62:38595 recv error: Network error: failed to read from TLS socket (remote: 127.28.112.62:38595): Connection reset by peer (error 104)
2026-08-09T07:57:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1511 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:43 dist-test-slave-vpvm krb5kdc[29794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:43 dist-test-slave-vpvm krb5kdc[29794](info): set up 2 sockets
Aug 09 07:57:43 dist-test-slave-vpvm krb5kdc[29794](info): commencing operation
krb5kdc: starting...
W20260809 07:57:45.065315 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:45 dist-test-slave-vpvm krb5kdc[29794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:57:45.184154 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:33757
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:33757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:45.290887 29819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:45.291123 29819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:45.291179 29819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:45.295013 29819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:45.295063 29819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:45.295089 29819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:45.295109 29819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:45.295131 29819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:45.300084 29819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:33757
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:33757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:45.300992 29819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:45.301915 29819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:45.309851 29827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:45.309908 29830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:45.309851 29826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:45.310053 29819 server_base.cc:1034] running on GCE node
I20260809 07:57:45.310670 29819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:45.311671 29819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:45.312841 29819 hybrid_clock.cc:648] HybridClock initialized: now 1786262265312814 us; error 37 us; skew 500 ppm
Aug 09 07:57:45 dist-test-slave-vpvm krb5kdc[29794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262265, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:45.315454 29819 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:57:45.316316 29819 webserver.cc:460] Webserver started at http://127.28.112.62:46085/ using document root <none> and password file <none>
I20260809 07:57:45.316845 29819 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:45.316910 29819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:45.317114 29819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:45.319149 29819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "cfa3b7f569d640368bd63338b04f526d"
format_stamp: "Formatted at 2026-08-09 07:57:45 on dist-test-slave-vpvm"
server_key: "17d706e89a48d910a103364282e3823b"
server_key_iv: "90a52b3dffba021458d7879dd4bfa024"
server_key_version: "encryptionkey@0"
I20260809 07:57:45.319689 29819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "cfa3b7f569d640368bd63338b04f526d"
format_stamp: "Formatted at 2026-08-09 07:57:45 on dist-test-slave-vpvm"
server_key: "17d706e89a48d910a103364282e3823b"
server_key_iv: "90a52b3dffba021458d7879dd4bfa024"
server_key_version: "encryptionkey@0"
I20260809 07:57:45.323239 29819 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:57:45.325480 29845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:45.326661 29819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:57:45.326777 29819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "cfa3b7f569d640368bd63338b04f526d"
format_stamp: "Formatted at 2026-08-09 07:57:45 on dist-test-slave-vpvm"
server_key: "17d706e89a48d910a103364282e3823b"
server_key_iv: "90a52b3dffba021458d7879dd4bfa024"
server_key_version: "encryptionkey@0"
I20260809 07:57:45.326867 29819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:45.345402 29819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:45.348093 29819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:45.348276 29819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:45.354857 29819 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:33757
I20260809 07:57:45.354899 29924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:33757 every 8 connection(s)
I20260809 07:57:45.355786 29819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:45.358500 29926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:45.359725 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 29819
I20260809 07:57:45.359853 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:45.360131 29120 external_mini_cluster.cc:1428] Setting key 3dfd2cc2b062f33a8b291c68a8c9a811
I20260809 07:57:45.364372 29926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d: Bootstrap starting.
I20260809 07:57:45.366930 29926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:45.367636 29926 log.cc:824] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:45.368733 29929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:45.361461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51441 (local address 127.28.112.62:33757)
0809 07:57:45.361858 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:45.361873 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:45.361908 (+ 35us) server_negotiation.cc:407] Connection header received
0809 07:57:45.362578 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:45.362596 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:45.362926 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:45.363223 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:45.363542 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:45.364478 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:45.365643 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:45.366129 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:45.366244 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:45.367110 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:45.367115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:45.367128 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:45.367488 (+ 360us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:45.367540 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:45.367561 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:45.367668 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:45.367890 (+ 222us) server_negotiation.cc:299] Negotiation successful
0809 07:57:45.368029 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":109,"threads_started":1}
I20260809 07:57:45.369451 29926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d: No bootstrap required, opened a new log
I20260809 07:57:45.371374 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:57:45.373024 29926 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa3b7f569d640368bd63338b04f526d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 33757 } }
I20260809 07:57:45.373329 29926 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:45.373401 29926 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfa3b7f569d640368bd63338b04f526d, State: Initialized, Role: FOLLOWER
I20260809 07:57:45.373843 29926 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa3b7f569d640368bd63338b04f526d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 33757 } }
I20260809 07:57:45.373991 29926 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:45.374058 29926 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:45.374156 29926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:45.374948 29926 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa3b7f569d640368bd63338b04f526d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 33757 } }
I20260809 07:57:45.375259 29926 leader_election.cc:302] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfa3b7f569d640368bd63338b04f526d; no voters:
I20260809 07:57:45.375548 29926 leader_election.cc:288] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:45.375653 29938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:45.375862 29938 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d [term 1 LEADER]: Becoming Leader. State: Replica: cfa3b7f569d640368bd63338b04f526d, State: Running, Role: LEADER
I20260809 07:57:45.376230 29938 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa3b7f569d640368bd63338b04f526d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 33757 } }
I20260809 07:57:45.376647 29926 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:45.377348 29929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:45.373244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55679 (local address 127.28.112.62:33757)
0809 07:57:45.373346 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:45.373349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:45.373445 (+ 96us) server_negotiation.cc:407] Connection header received
0809 07:57:45.373603 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:45.373606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:45.373644 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:45.373716 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:45.373999 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:45.374594 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:45.375469 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:45.375798 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:45.375902 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:45.376854 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:45.376857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:45.376860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:45.376872 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:45.376899 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:45.376911 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:45.376992 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:45.377191 (+ 199us) server_negotiation.cc:299] Negotiation successful
0809 07:57:45.377226 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 07:57:45.377879 29938 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfa3b7f569d640368bd63338b04f526d. Latest consensus state: current_term: 1 leader_uuid: "cfa3b7f569d640368bd63338b04f526d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa3b7f569d640368bd63338b04f526d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 33757 } } }
I20260809 07:57:45.378093 29938 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:45.378532 29953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:45.378731 29940 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfa3b7f569d640368bd63338b04f526d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa3b7f569d640368bd63338b04f526d" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 33757 } } }
I20260809 07:57:45.378932 29940 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:45.381645 29953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:45.386703 29953 catalog_manager.cc:1347] Generated new cluster ID: 642fd32e33d1443993cfbf7a9b54a692
I20260809 07:57:45.386763 29953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:45.393021 29953 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:45.393838 29953 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:45.406262 29953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfa3b7f569d640368bd63338b04f526d: Generated new TSK 0
I20260809 07:57:45.406889 29953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:45.412552 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 29819
W20260809 07:57:45.431561 29791 connection.cc:535] client connection to 127.28.112.62:33757 recv error: Network error: failed to read from TLS socket (remote: 127.28.112.62:33757): Connection reset by peer (error 104)
2026-08-09T07:57:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1482 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:45 dist-test-slave-vpvm krb5kdc[29967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:45 dist-test-slave-vpvm krb5kdc[29967](info): set up 2 sockets
Aug 09 07:57:45 dist-test-slave-vpvm krb5kdc[29967](info): commencing operation
krb5kdc: starting...
W20260809 07:57:46.567149 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:46 dist-test-slave-vpvm krb5kdc[29967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:57:46.681838 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:39381
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:46871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:39381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:46.789086 29988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:46.789319 29988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:46.789361 29988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:46.793217 29988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:46.793272 29988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:46.793298 29988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:46.793322 29988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:46.793339 29988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:46.798276 29988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:46871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:39381
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:39381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:46.799149 29988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:46.800083 29988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:46.808045 29995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:46.808073 29996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:46.808152 29988 server_base.cc:1034] running on GCE node
W20260809 07:57:46.808092 29999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:46.808790 29988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:46.809813 29988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:46.810969 29988 hybrid_clock.cc:648] HybridClock initialized: now 1786262266810952 us; error 33 us; skew 500 ppm
Aug 09 07:57:46 dist-test-slave-vpvm krb5kdc[29967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262266, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:46.813488 29988 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:57:46.814361 29988 webserver.cc:460] Webserver started at http://127.28.112.62:36205/ using document root <none> and password file <none>
I20260809 07:57:46.814843 29988 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:46.814898 29988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:46.815058 29988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:46.817026 29988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "42d125652dc345cb9fa35ffefcc4f090"
format_stamp: "Formatted at 2026-08-09 07:57:46 on dist-test-slave-vpvm"
server_key: "b3cdaa57267495b8a8ab0d83049458ee"
server_key_iv: "a73a264f1da6b13ad6d232e889fb98f7"
server_key_version: "encryptionkey@0"
I20260809 07:57:46.817523 29988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "42d125652dc345cb9fa35ffefcc4f090"
format_stamp: "Formatted at 2026-08-09 07:57:46 on dist-test-slave-vpvm"
server_key: "b3cdaa57267495b8a8ab0d83049458ee"
server_key_iv: "a73a264f1da6b13ad6d232e889fb98f7"
server_key_version: "encryptionkey@0"
I20260809 07:57:46.820945 29988 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:57:46.823242 30008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:46.824383 29988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 07:57:46.824483 29988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "42d125652dc345cb9fa35ffefcc4f090"
format_stamp: "Formatted at 2026-08-09 07:57:46 on dist-test-slave-vpvm"
server_key: "b3cdaa57267495b8a8ab0d83049458ee"
server_key_iv: "a73a264f1da6b13ad6d232e889fb98f7"
server_key_version: "encryptionkey@0"
I20260809 07:57:46.824563 29988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:46.828848 29988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:46.831440 29988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:46.831576 29988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:46.837921 29988 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:39381
I20260809 07:57:46.837932 30090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:39381 every 8 connection(s)
I20260809 07:57:46.838847 29988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:46.841655 30092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:46.846640 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 29988
I20260809 07:57:46.846786 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:46.847080 29120 external_mini_cluster.cc:1428] Setting key 99e7807d0c5ebf92828127a92ebe72c4
I20260809 07:57:46.847568 30092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090: Bootstrap starting.
I20260809 07:57:46.850013 30092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:46.850807 30092 log.cc:824] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:46.852243 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:46.848513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48447 (local address 127.28.112.62:39381)
0809 07:57:46.848951 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:46.848966 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:46.849002 (+ 36us) server_negotiation.cc:407] Connection header received
0809 07:57:46.849623 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:46.849641 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:46.849914 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:46.850286 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:46.851028 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:46.851034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:46.851049 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:46.851063 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:46.851131 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:46.851176 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:46.851284 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:46.851401 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 07:57:46.851542 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":132,"threads_started":1}
I20260809 07:57:46.852730 30092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090: No bootstrap required, opened a new log
I20260809 07:57:46.854810 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:57:46.856601 30092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d125652dc345cb9fa35ffefcc4f090" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39381 } }
I20260809 07:57:46.856861 30092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:46.856976 30092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d125652dc345cb9fa35ffefcc4f090, State: Initialized, Role: FOLLOWER
I20260809 07:57:46.857429 30092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d125652dc345cb9fa35ffefcc4f090" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39381 } }
I20260809 07:57:46.857605 30092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:46.857681 30092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:46.857781 30092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:46.858546 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:46.857051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38765 (local address 127.28.112.62:39381)
0809 07:57:46.857179 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:46.857183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:46.857358 (+ 175us) server_negotiation.cc:407] Connection header received
0809 07:57:46.857448 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:46.857451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:46.857485 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:46.857567 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:46.858233 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:46.858237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:46.858239 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:46.858250 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:46.858276 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:46.858289 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:46.858375 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:46.858406 (+ 31us) server_negotiation.cc:299] Negotiation successful
0809 07:57:46.858439 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 07:57:46.858824 30092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d125652dc345cb9fa35ffefcc4f090" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39381 } }
I20260809 07:57:46.859148 30092 leader_election.cc:302] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42d125652dc345cb9fa35ffefcc4f090; no voters:
I20260809 07:57:46.859408 30092 leader_election.cc:288] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:46.859493 30105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:46.859701 30105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090 [term 1 LEADER]: Becoming Leader. State: Replica: 42d125652dc345cb9fa35ffefcc4f090, State: Running, Role: LEADER
I20260809 07:57:46.860045 30105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d125652dc345cb9fa35ffefcc4f090" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39381 } }
I20260809 07:57:46.860538 30092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:46.861745 30107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42d125652dc345cb9fa35ffefcc4f090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d125652dc345cb9fa35ffefcc4f090" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39381 } } }
I20260809 07:57:46.861754 30109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42d125652dc345cb9fa35ffefcc4f090. Latest consensus state: current_term: 1 leader_uuid: "42d125652dc345cb9fa35ffefcc4f090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d125652dc345cb9fa35ffefcc4f090" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39381 } } }
I20260809 07:57:46.861857 30107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:46.861865 30109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:46.862318 30120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:46.865156 30120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:46.870409 30120 catalog_manager.cc:1347] Generated new cluster ID: cb841c22524447f5b0962aba52dd241f
I20260809 07:57:46.870482 30120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:46.878891 30120 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:46.879179 30120 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:46.883543 30120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42d125652dc345cb9fa35ffefcc4f090: Generated new TSK 0
I20260809 07:57:46.884121 30120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:46.886976 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 29988
2026-08-09T07:57:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1471 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:46 dist-test-slave-vpvm krb5kdc[30133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:46 dist-test-slave-vpvm krb5kdc[30133](info): set up 2 sockets
Aug 09 07:57:46 dist-test-slave-vpvm krb5kdc[30133](info): commencing operation
krb5kdc: starting...
W20260809 07:57:48.021698 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:48 dist-test-slave-vpvm krb5kdc[30133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:57:48.137010 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:40221
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:40221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:48.245644 30162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:48.245896 30162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:48.245936 30162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:48.250487 30162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:48.250558 30162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:48.250586 30162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:48.250608 30162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:48.250644 30162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:48.257426 30162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:40221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:40221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:48.258497 30162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:48.259475 30162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:48.267710 30169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:48.267683 30168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:48.267692 30172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:48.268185 30162 server_base.cc:1034] running on GCE node
I20260809 07:57:48.268705 30162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:48.269757 30162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:48.270934 30162 hybrid_clock.cc:648] HybridClock initialized: now 1786262268270911 us; error 36 us; skew 500 ppm
Aug 09 07:57:48 dist-test-slave-vpvm krb5kdc[30133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262268, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:48.273546 30162 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:57:48.274434 30162 webserver.cc:460] Webserver started at http://127.28.112.62:44367/ using document root <none> and password file <none>
I20260809 07:57:48.274957 30162 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:48.275022 30162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:48.275224 30162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:48.277194 30162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "2d70dadfa3d44bcd8b7ebb83ba1a8e13"
format_stamp: "Formatted at 2026-08-09 07:57:48 on dist-test-slave-vpvm"
server_key: "5481ebd4b323f31a9b95af3d82a1393c"
server_key_iv: "b0bda70efff87c33b14093fdf19f5120"
server_key_version: "encryptionkey@0"
I20260809 07:57:48.277714 30162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "2d70dadfa3d44bcd8b7ebb83ba1a8e13"
format_stamp: "Formatted at 2026-08-09 07:57:48 on dist-test-slave-vpvm"
server_key: "5481ebd4b323f31a9b95af3d82a1393c"
server_key_iv: "b0bda70efff87c33b14093fdf19f5120"
server_key_version: "encryptionkey@0"
I20260809 07:57:48.281195 30162 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:57:48.283455 30181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:48.284570 30162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260809 07:57:48.284686 30162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "2d70dadfa3d44bcd8b7ebb83ba1a8e13"
format_stamp: "Formatted at 2026-08-09 07:57:48 on dist-test-slave-vpvm"
server_key: "5481ebd4b323f31a9b95af3d82a1393c"
server_key_iv: "b0bda70efff87c33b14093fdf19f5120"
server_key_version: "encryptionkey@0"
I20260809 07:57:48.284776 30162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:48.316140 30162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:48.319103 30162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:48.319289 30162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:48.325974 30162 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:40221
I20260809 07:57:48.326020 30261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:40221 every 8 connection(s)
I20260809 07:57:48.326948 30162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:48.329718 30263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:48.333060 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 30162
I20260809 07:57:48.333220 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:48.333508 29120 external_mini_cluster.cc:1428] Setting key 7eabc1fe9909d930b1bf8517a88b1316
I20260809 07:57:48.335870 30263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13: Bootstrap starting.
I20260809 07:57:48.338410 30263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:48.339283 30263 log.cc:824] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:48.341542 30263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13: No bootstrap required, opened a new log
Aug 09 07:57:48 dist-test-slave-vpvm krb5kdc[30133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:57:48.344907 30263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d70dadfa3d44bcd8b7ebb83ba1a8e13" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40221 } }
I20260809 07:57:48.345127 30263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:48.345172 30263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d70dadfa3d44bcd8b7ebb83ba1a8e13, State: Initialized, Role: FOLLOWER
I20260809 07:57:48.345605 30263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d70dadfa3d44bcd8b7ebb83ba1a8e13" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40221 } }
I20260809 07:57:48.345772 30263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:48.345841 30263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:48.345968 30263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:48.346966 30263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d70dadfa3d44bcd8b7ebb83ba1a8e13" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40221 } }
I20260809 07:57:48.347352 30263 leader_election.cc:302] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d70dadfa3d44bcd8b7ebb83ba1a8e13; no voters:
I20260809 07:57:48.347764 30263 leader_election.cc:288] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:48.347841 30271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:48.348050 30271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13 [term 1 LEADER]: Becoming Leader. State: Replica: 2d70dadfa3d44bcd8b7ebb83ba1a8e13, State: Running, Role: LEADER
I20260809 07:57:48.348429 30271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d70dadfa3d44bcd8b7ebb83ba1a8e13" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40221 } }
I20260809 07:57:48.348587 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:48.334943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53339 (local address 127.28.112.62:40221)
0809 07:57:48.335356 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:48.335374 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:48.335412 (+ 38us) server_negotiation.cc:407] Connection header received
0809 07:57:48.336075 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:48.336096 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:48.336460 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:48.336748 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:48.337625 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:48.338694 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:48.339828 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:48.340395 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:48.340519 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:48.343343 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:48.343348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:48.343363 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:48.343391 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:48.345533 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:48.346063 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:48.346070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:48.346076 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:48.346200 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:48.346478 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:48.346480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:48.346481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:48.346890 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:48.347053 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:48.347371 (+ 318us) server_negotiation.cc:299] Negotiation successful
0809 07:57:48.347656 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":105,"threads_started":1}
I20260809 07:57:48.349140 30263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:48.351109 30272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d70dadfa3d44bcd8b7ebb83ba1a8e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d70dadfa3d44bcd8b7ebb83ba1a8e13" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40221 } } }
I20260809 07:57:48.351234 30272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:48.351780 30273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d70dadfa3d44bcd8b7ebb83ba1a8e13. Latest consensus state: current_term: 1 leader_uuid: "2d70dadfa3d44bcd8b7ebb83ba1a8e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d70dadfa3d44bcd8b7ebb83ba1a8e13" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40221 } } }
I20260809 07:57:48.351882 30273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:48.352100 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260809 07:57:48.354771 30290 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:57:48.354853 30290 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:57:48.354912 30291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:48.355789 30291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:48.361584 30291 catalog_manager.cc:1347] Generated new cluster ID: e654dbcc2e5c4543bd5425661de97e60
I20260809 07:57:48.361645 30291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:48.366137 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:48.353976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58203 (local address 127.28.112.62:40221)
0809 07:57:48.354086 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:48.354090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:48.358247 (+ 4157us) server_negotiation.cc:407] Connection header received
0809 07:57:48.358364 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:48.358367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:48.358414 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:48.358497 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:48.359364 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:48.359974 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:48.360742 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:48.360958 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:48.361043 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:48.362966 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:48.362970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:48.362973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:48.363000 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:48.364551 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:48.364976 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:48.364980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:48.364982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:48.365070 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:48.365308 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:48.365311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:48.365313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:48.365525 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:48.365661 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:48.365842 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 07:57:48.365982 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 07:57:48.387774 30291 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:48.388744 30291 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:48.395512 30291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d70dadfa3d44bcd8b7ebb83ba1a8e13: Generated new TSK 0
I20260809 07:57:48.396191 30291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:48.401058 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 30162
2026-08-09T07:57:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1509 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:48 dist-test-slave-vpvm krb5kdc[30315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:48 dist-test-slave-vpvm krb5kdc[30315](info): set up 2 sockets
Aug 09 07:57:48 dist-test-slave-vpvm krb5kdc[30315](info): commencing operation
krb5kdc: starting...
W20260809 07:57:49.542788 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:49 dist-test-slave-vpvm krb5kdc[30315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:57:49.661331 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:41539
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:41539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:49.770489 30333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:49.770733 30333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:49.770773 30333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:49.774619 30333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:49.774667 30333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:49.774685 30333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:49.774699 30333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:49.774720 30333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:49.779850 30333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:41539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:41539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:49.780735 30333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:49.781625 30333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:49.789574 30340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:49.789609 30343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:49.789646 30339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:49.789809 30333 server_base.cc:1034] running on GCE node
I20260809 07:57:49.790501 30333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:49.791538 30333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:49.792702 30333 hybrid_clock.cc:648] HybridClock initialized: now 1786262269792682 us; error 30 us; skew 500 ppm
Aug 09 07:57:49 dist-test-slave-vpvm krb5kdc[30315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262269, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:49.795624 30333 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:57:49.796460 30333 webserver.cc:460] Webserver started at http://127.28.112.62:32805/ using document root <none> and password file <none>
I20260809 07:57:49.796988 30333 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:49.797053 30333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:49.797240 30333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:49.799232 30333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "262ca888a79c419aba809d7a3b23fa7a"
format_stamp: "Formatted at 2026-08-09 07:57:49 on dist-test-slave-vpvm"
server_key: "3fd57afcaa6189047ac2d7e62146bd4b"
server_key_iv: "9427f36f81a0b291c4fb2e3060550a24"
server_key_version: "encryptionkey@0"
I20260809 07:57:49.799760 30333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "262ca888a79c419aba809d7a3b23fa7a"
format_stamp: "Formatted at 2026-08-09 07:57:49 on dist-test-slave-vpvm"
server_key: "3fd57afcaa6189047ac2d7e62146bd4b"
server_key_iv: "9427f36f81a0b291c4fb2e3060550a24"
server_key_version: "encryptionkey@0"
I20260809 07:57:49.803283 30333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:57:49.805485 30359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:49.806653 30333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 07:57:49.806770 30333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "262ca888a79c419aba809d7a3b23fa7a"
format_stamp: "Formatted at 2026-08-09 07:57:49 on dist-test-slave-vpvm"
server_key: "3fd57afcaa6189047ac2d7e62146bd4b"
server_key_iv: "9427f36f81a0b291c4fb2e3060550a24"
server_key_version: "encryptionkey@0"
I20260809 07:57:49.806859 30333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:49.833124 30333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:49.836283 30333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:49.836470 30333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:49.843258 30333 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:41539
I20260809 07:57:49.843267 30433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:41539 every 8 connection(s)
I20260809 07:57:49.844236 30333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:49.846906 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 30333
I20260809 07:57:49.847025 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:49.847016 30437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:49.847326 29120 external_mini_cluster.cc:1428] Setting key 15ff50d6804ba32e50e8fdcc0b6c9761
I20260809 07:57:49.853915 30437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a: Bootstrap starting.
I20260809 07:57:49.856470 30437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:49.857242 30442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:49.848693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57203 (local address 127.28.112.62:41539)
0809 07:57:49.849115 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:49.849138 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:49.849181 (+ 43us) server_negotiation.cc:407] Connection header received
0809 07:57:49.850232 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:49.850256 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:49.850686 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:49.851045 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:49.851444 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:49.852911 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:49.853745 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:49.854449 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:49.854601 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:49.855350 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:49.855356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:49.855371 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:49.855832 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:49.855901 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:49.855931 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:49.856028 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:49.856124 (+ 96us) server_negotiation.cc:299] Negotiation successful
0809 07:57:49.856325 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":112,"threads_started":1}
I20260809 07:57:49.857329 30437 log.cc:824] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:49.859349 30437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a: No bootstrap required, opened a new log
I20260809 07:57:49.859925 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:57:49.863116 30437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262ca888a79c419aba809d7a3b23fa7a" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 41539 } }
I20260809 07:57:49.863394 30437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:49.863468 30437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 262ca888a79c419aba809d7a3b23fa7a, State: Initialized, Role: FOLLOWER
I20260809 07:57:49.863925 30437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262ca888a79c419aba809d7a3b23fa7a" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 41539 } }
I20260809 07:57:49.864091 30437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:49.864159 30437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:49.864255 30437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:49.865312 30437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262ca888a79c419aba809d7a3b23fa7a" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 41539 } }
I20260809 07:57:49.865502 30442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:49.861764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57965 (local address 127.28.112.62:41539)
0809 07:57:49.861907 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:49.861910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:49.862038 (+ 128us) server_negotiation.cc:407] Connection header received
0809 07:57:49.862152 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:49.862155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:49.862189 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:49.862249 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:49.862514 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:49.862981 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:49.864020 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:49.864341 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:49.864427 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:49.865134 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:49.865137 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:49.865139 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:49.865150 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:49.865172 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:49.865181 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:49.865235 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:49.865341 (+ 106us) server_negotiation.cc:299] Negotiation successful
0809 07:57:49.865380 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 07:57:49.865681 30437 leader_election.cc:302] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 262ca888a79c419aba809d7a3b23fa7a; no voters:
I20260809 07:57:49.866014 30437 leader_election.cc:288] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:49.866081 30451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:49.866321 30451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a [term 1 LEADER]: Becoming Leader. State: Replica: 262ca888a79c419aba809d7a3b23fa7a, State: Running, Role: LEADER
I20260809 07:57:49.866772 30451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262ca888a79c419aba809d7a3b23fa7a" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 41539 } }
I20260809 07:57:49.867211 30437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:49.868266 30453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 262ca888a79c419aba809d7a3b23fa7a. Latest consensus state: current_term: 1 leader_uuid: "262ca888a79c419aba809d7a3b23fa7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262ca888a79c419aba809d7a3b23fa7a" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 41539 } } }
I20260809 07:57:49.868393 30453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:49.868724 30452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "262ca888a79c419aba809d7a3b23fa7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262ca888a79c419aba809d7a3b23fa7a" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 41539 } } }
I20260809 07:57:49.868813 30452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:49.868795 30460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:49.871598 30460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:49.876600 30460 catalog_manager.cc:1347] Generated new cluster ID: 31302ea3e6254c12abd252e3290aa149
I20260809 07:57:49.876655 30460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:49.888090 30460 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:49.889254 30460 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:49.898973 30460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 262ca888a79c419aba809d7a3b23fa7a: Generated new TSK 0
I20260809 07:57:49.899546 30460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:49.905485 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 30333
2026-08-09T07:57:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1509 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:49 dist-test-slave-vpvm krb5kdc[30479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:49 dist-test-slave-vpvm krb5kdc[30479](info): set up 2 sockets
Aug 09 07:57:49 dist-test-slave-vpvm krb5kdc[30479](info): commencing operation
krb5kdc: starting...
W20260809 07:57:51.048421 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:51 dist-test-slave-vpvm krb5kdc[30479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:57:51.164515 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:41115
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:41115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:51.280704 30495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:51.281030 30495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:51.281129 30495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:51.285426 30495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:51.285491 30495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:51.285509 30495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:51.285526 30495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:51.285539 30495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:51.290827 30495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:41115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:41115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:51.291838 30495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:51.292814 30495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:51.300840 30504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:51.300877 30502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:51.300882 30510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:51.300976 30495 server_base.cc:1034] running on GCE node
I20260809 07:57:51.301664 30495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:51.302727 30495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:51.303884 30495 hybrid_clock.cc:648] HybridClock initialized: now 1786262271303867 us; error 31 us; skew 500 ppm
Aug 09 07:57:51 dist-test-slave-vpvm krb5kdc[30479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262271, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:51.306460 30495 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:57:51.307298 30495 webserver.cc:460] Webserver started at http://127.28.112.62:39727/ using document root <none> and password file <none>
I20260809 07:57:51.307797 30495 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:51.307853 30495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:51.308017 30495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:51.309899 30495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "f117d78e0a324902982359c08c5ed862"
format_stamp: "Formatted at 2026-08-09 07:57:51 on dist-test-slave-vpvm"
server_key: "7e3565f94f61ea202a9045876b689180"
server_key_iv: "4debef7f60397d1f4a47751482a8adb2"
server_key_version: "encryptionkey@0"
I20260809 07:57:51.310431 30495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "f117d78e0a324902982359c08c5ed862"
format_stamp: "Formatted at 2026-08-09 07:57:51 on dist-test-slave-vpvm"
server_key: "7e3565f94f61ea202a9045876b689180"
server_key_iv: "4debef7f60397d1f4a47751482a8adb2"
server_key_version: "encryptionkey@0"
I20260809 07:57:51.313813 30495 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 07:57:51.316043 30523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:51.317205 30495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:57:51.317322 30495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "f117d78e0a324902982359c08c5ed862"
format_stamp: "Formatted at 2026-08-09 07:57:51 on dist-test-slave-vpvm"
server_key: "7e3565f94f61ea202a9045876b689180"
server_key_iv: "4debef7f60397d1f4a47751482a8adb2"
server_key_version: "encryptionkey@0"
I20260809 07:57:51.317413 30495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:51.321863 30495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:51.324554 30495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:51.324725 30495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:51.331513 30495 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:41115
I20260809 07:57:51.331523 30597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:41115 every 8 connection(s)
I20260809 07:57:51.332427 30495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:51.335255 30599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:51.340917 30599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862: Bootstrap starting.
I20260809 07:57:51.342202 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 30495
I20260809 07:57:51.342336 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:51.342609 29120 external_mini_cluster.cc:1428] Setting key 541f4fd3654bc00a00ba6fad4142bbaa
I20260809 07:57:51.343859 30599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:51.344607 30599 log.cc:824] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:51.346869 30599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862: No bootstrap required, opened a new log
I20260809 07:57:51.347587 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:51.343746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40301 (local address 127.28.112.62:41115)
0809 07:57:51.344167 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:51.344182 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:51.344219 (+ 37us) server_negotiation.cc:407] Connection header received
0809 07:57:51.344883 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:51.344903 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:51.345263 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:51.345588 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:51.346386 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:51.346391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:51.346403 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:51.346412 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:51.346464 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:51.346483 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:51.346567 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:51.346699 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 07:57:51.346847 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
I20260809 07:57:51.350431 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:57:51.350879 30599 raft_consensus.cc:348] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f117d78e0a324902982359c08c5ed862" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 41115 } }
I20260809 07:57:51.351099 30599 raft_consensus.cc:374] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 07:57:51.351166 30599 raft_consensus.cc:729] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f117d78e0a324902982359c08c5ed862, State: Initialized, Role: FOLLOWER
I20260809 07:57:51.351624 30599 consensus_queue.cc:260] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f117d78e0a324902982359c08c5ed862" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 41115 } }
I20260809 07:57:51.351791 30599 raft_consensus.cc:388] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 07:57:51.351859 30599 raft_consensus.cc:482] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 07:57:51.351955 30599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:51.353018 30599 raft_consensus.cc:504] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f117d78e0a324902982359c08c5ed862" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 41115 } }
I20260809 07:57:51.353346 30599 leader_election.cc:302] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f117d78e0a324902982359c08c5ed862; no voters:
I20260809 07:57:51.353653 30599 leader_election.cc:288] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 07:57:51.353742 30610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:51.353904 30610 raft_consensus.cc:686] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862 [term 1 LEADER]: Becoming Leader. State: Replica: f117d78e0a324902982359c08c5ed862, State: Running, Role: LEADER
I20260809 07:57:51.354132 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:51.352441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45171 (local address 127.28.112.62:41115)
0809 07:57:51.352543 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:51.352548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:51.352751 (+ 203us) server_negotiation.cc:407] Connection header received
0809 07:57:51.352862 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:51.352865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:51.352897 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:51.352963 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:51.353660 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:51.353664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:51.353667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 07:57:51.353680 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:51.353708 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 07:57:51.353721 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:51.353794 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:51.353937 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 07:57:51.354011 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 07:57:51.354290 30610 consensus_queue.cc:237] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f117d78e0a324902982359c08c5ed862" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 41115 } }
I20260809 07:57:51.355013 30599 sys_catalog.cc:564] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:51.355580 30610 sys_catalog.cc:455] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f117d78e0a324902982359c08c5ed862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f117d78e0a324902982359c08c5ed862" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 41115 } } }
I20260809 07:57:51.355697 30610 sys_catalog.cc:458] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:51.355962 30610 sys_catalog.cc:455] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f117d78e0a324902982359c08c5ed862. Latest consensus state: current_term: 1 leader_uuid: "f117d78e0a324902982359c08c5ed862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f117d78e0a324902982359c08c5ed862" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 41115 } } }
I20260809 07:57:51.356107 30619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:51.356115 30610 sys_catalog.cc:458] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:51.359108 30619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:51.364748 30619 catalog_manager.cc:1347] Generated new cluster ID: 5292e867e66744b7932ff6c2887c140a
I20260809 07:57:51.364815 30619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:51.374814 30619 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:51.375093 30619 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:51.382903 30619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f117d78e0a324902982359c08c5ed862: Generated new TSK 0
I20260809 07:57:51.383502 30619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 07:57:51.389622 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 30495
2026-08-09T07:57:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1477 ms)
[----------] 18 tests from AuthTokenIssuingTest (27008 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:51 dist-test-slave-vpvm krb5kdc[30641](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:51 dist-test-slave-vpvm krb5kdc[30641](info): set up 2 sockets
Aug 09 07:57:51 dist-test-slave-vpvm krb5kdc[30641](info): commencing operation
krb5kdc: starting...
W20260809 07:57:52.524139 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:52 dist-test-slave-vpvm krb5kdc[30641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:57:52.642459 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:40725
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:40725,127.28.112.61:38417,127.28.112.60:38561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:52.800156 30663 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 07:57:52.800771 30663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:52.800912 30663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:52.801064 30663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:52.807551 30663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:52.807677 30663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:52.807754 30663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:52.807819 30663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:52.807914 30663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:52.816598 30663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:40725,127.28.112.61:38417,127.28.112.60:38561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:40725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:52.818692 30663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:52.820712 30663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:52.835497 30668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:52.835493 30671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:52.835522 30669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:52.835994 30663 server_base.cc:1034] running on GCE node
I20260809 07:57:52.837023 30663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:52.839073 30663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:52.840297 30663 hybrid_clock.cc:648] HybridClock initialized: now 1786262272840266 us; error 51 us; skew 500 ppm
Aug 09 07:57:52 dist-test-slave-vpvm krb5kdc[30641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262272, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:52.844821 30663 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:57:52.846362 30663 webserver.cc:460] Webserver started at http://127.28.112.62:46623/ using document root <none> and password file <none>
I20260809 07:57:52.847393 30663 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:52.847514 30663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:52.847893 30663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:52.851771 30663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "15bf553506aa4188b5aee880eec43ded"
format_stamp: "Formatted at 2026-08-09 07:57:52 on dist-test-slave-vpvm"
server_key: "a001513e3923a2f8f9daa6127fe2e5cc"
server_key_iv: "b2af79f2afcbc31a67524ccb541b1d91"
server_key_version: "encryptionkey@0"
I20260809 07:57:52.852716 30663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "15bf553506aa4188b5aee880eec43ded"
format_stamp: "Formatted at 2026-08-09 07:57:52 on dist-test-slave-vpvm"
server_key: "a001513e3923a2f8f9daa6127fe2e5cc"
server_key_iv: "b2af79f2afcbc31a67524ccb541b1d91"
server_key_version: "encryptionkey@0"
I20260809 07:57:52.859428 30663 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 07:57:52.863593 30685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:52.865579 30663 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 07:57:52.865787 30663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "15bf553506aa4188b5aee880eec43ded"
format_stamp: "Formatted at 2026-08-09 07:57:52 on dist-test-slave-vpvm"
server_key: "a001513e3923a2f8f9daa6127fe2e5cc"
server_key_iv: "b2af79f2afcbc31a67524ccb541b1d91"
server_key_version: "encryptionkey@0"
I20260809 07:57:52.865942 30663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:52.897631 30663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:52.902305 30663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:52.902606 30663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:52.914103 30663 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:40725
I20260809 07:57:52.914534 30760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:40725 every 8 connection(s)
I20260809 07:57:52.915781 30663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:52.920178 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 30663
I20260809 07:57:52.920378 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:52.920796 30764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:52.920868 29120 external_mini_cluster.cc:1428] Setting key 8a2b7b14130988d2d3f08c3855c8cfe6
I20260809 07:57:52.929822 30764 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.112.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.61@KRBTEST.COM" created.
Aug 09 07:57:52 dist-test-slave-vpvm krb5kdc[30641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262272, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:57:52.956843 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:52.934950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43462 (local address 127.28.112.62:40725)
0809 07:57:52.935970 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:52.935996 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:52.936106 (+ 110us) server_negotiation.cc:407] Connection header received
0809 07:57:52.937932 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:52.937938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:52.938365 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:52.938647 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:52.940016 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:52.941665 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:52.943897 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:52.944941 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:52.945260 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:52.949490 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:52.949504 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:52.949538 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:52.949583 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:52.951915 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:52.952595 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:52.952600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:52.952601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:52.952763 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:52.953255 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:52.953260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:52.953261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:52.953863 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:52.954034 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:52.954575 (+ 541us) server_negotiation.cc:299] Negotiation successful
0809 07:57:52.954995 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":795,"spinlock_wait_cycles":6272,"thread_start_us":333,"threads_started":1}
I20260809 07:57:52.956843 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:52.934949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:40725 (local address 127.0.0.1:43462)
0809 07:57:52.935826 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:52.935928 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:52.937448 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:52.938687 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:52.938694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:52.939261 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:52.939741 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:52.939768 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:52.941860 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:52.941865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:52.943739 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:52.943752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:52.945186 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:52.945198 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:52.945352 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:52.946354 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:52.946401 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:52.949249 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:52.952059 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:52.952070 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:52.952108 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:52.952439 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:52.952910 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:52.952925 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:52.952927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:52.953118 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:52.954016 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:52.954022 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:52.954237 (+ 215us) client_negotiation.cc:769] Sending connection context
0809 07:57:52.954487 (+ 250us) client_negotiation.cc:240] Negotiation successful
0809 07:57:52.955137 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":181,"spinlock_wait_cycles":7936,"thread_start_us":215,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.61.keytab.
Entry for principal kudu/127.28.112.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.61.keytab.
I20260809 07:57:52.960848 30764 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } has no permanent_uuid. Determining permanent_uuid...
I20260809 07:57:52.961992 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.61:38417
--webserver_interface=127.28.112.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.61.keytab
--principal=kudu/127.28.112.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:40725,127.28.112.61:38417,127.28.112.60:38561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 07:57:52.962059 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:52.961544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:38417 (local address 127.0.0.1:44454)
0809 07:57:52.961733 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:52.961892 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260809 07:57:52.962463 30686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.112.61:38417: connect: Connection refused (error 111)
I20260809 07:57:52.963573 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:52.963309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:38417 (local address 127.0.0.1:44458)
0809 07:57:52.963449 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:52.963502 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 07:57:52.963723 30764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.61:38417: Network error: Client connection negotiation failed: client connection to 127.28.112.61:38417: connect: Connection refused (error 111)
I20260809 07:57:53.013087 30764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } attempt: 1
I20260809 07:57:53.014569 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.014143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:38417 (local address 127.0.0.1:44460)
0809 07:57:53.014357 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.014451 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 07:57:53.015640 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.015312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:38417 (local address 127.0.0.1:44464)
0809 07:57:53.015487 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.015555 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 07:57:53.015837 30764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.61:38417: Network error: Client connection negotiation failed: client connection to 127.28.112.61:38417: connect: Connection refused (error 111)
I20260809 07:57:53.084050 30764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } attempt: 2
I20260809 07:57:53.085222 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.084731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:38417 (local address 127.0.0.1:44476)
0809 07:57:53.084980 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.085092 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260809 07:57:53.086349 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.085930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:38417 (local address 127.0.0.1:44492)
0809 07:57:53.086151 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.086231 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260809 07:57:53.086516 30764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.61:38417: Network error: Client connection negotiation failed: client connection to 127.28.112.61:38417: connect: Connection refused (error 111)
W20260809 07:57:53.126432 30778 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 07:57:53.126896 30778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:53.126962 30778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:53.127101 30778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:53.133327 30778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:53.133424 30778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:53.133468 30778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:53.133526 30778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:53.133584 30778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:53.142041 30778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:40725,127.28.112.61:38417,127.28.112.60:38561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.61.keytab
--principal=kudu/127.28.112.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.61:38417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.112.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:53.143976 30778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:53.145766 30778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:53.159054 30791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:53.159132 30788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:53.159056 30789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:53.159723 30778 server_base.cc:1034] running on GCE node
I20260809 07:57:53.160693 30778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:53.162591 30778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:53.163918 30778 hybrid_clock.cc:648] HybridClock initialized: now 1786262273163875 us; error 67 us; skew 500 ppm
Aug 09 07:57:53 dist-test-slave-vpvm krb5kdc[30641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262273, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:53.168267 30778 init.cc:377] Logged in from keytab as kudu/127.28.112.61@KRBTEST.COM (short username kudu)
I20260809 07:57:53.169691 30778 webserver.cc:460] Webserver started at http://127.28.112.61:44725/ using document root <none> and password file <none>
I20260809 07:57:53.170794 30778 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:53.170885 30778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:53.171200 30778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:53.174535 30778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-1/data/instance:
uuid: "50ca5d0e7d02407880a4ee86229160b7"
format_stamp: "Formatted at 2026-08-09 07:57:53 on dist-test-slave-vpvm"
server_key: "b6ddb6d9e7c736f2996884758c43a40f"
server_key_iv: "6d4e928ff8462b764183e0e49c0f81f9"
server_key_version: "encryptionkey@0"
I20260809 07:57:53.175433 30778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-1/wal/instance:
uuid: "50ca5d0e7d02407880a4ee86229160b7"
format_stamp: "Formatted at 2026-08-09 07:57:53 on dist-test-slave-vpvm"
server_key: "b6ddb6d9e7c736f2996884758c43a40f"
server_key_iv: "6d4e928ff8462b764183e0e49c0f81f9"
server_key_version: "encryptionkey@0"
I20260809 07:57:53.177688 30764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } attempt: 3
I20260809 07:57:53.178891 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.178365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:38417 (local address 127.0.0.1:44506)
0809 07:57:53.178630 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.178737 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260809 07:57:53.180156 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.179678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:38417 (local address 127.0.0.1:44510)
0809 07:57:53.179893 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.180022 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 07:57:53.180367 30764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.61:38417: Network error: Client connection negotiation failed: client connection to 127.28.112.61:38417: connect: Connection refused (error 111)
I20260809 07:57:53.182643 30778 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260809 07:57:53.186753 30798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:53.188719 30778 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 07:57:53.188907 30778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-1/wal
uuid: "50ca5d0e7d02407880a4ee86229160b7"
format_stamp: "Formatted at 2026-08-09 07:57:53 on dist-test-slave-vpvm"
server_key: "b6ddb6d9e7c736f2996884758c43a40f"
server_key_iv: "6d4e928ff8462b764183e0e49c0f81f9"
server_key_version: "encryptionkey@0"
I20260809 07:57:53.189069 30778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:53.211508 30778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:53.215799 30778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:53.216063 30778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:53.227415 30778 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.61:38417
I20260809 07:57:53.227820 30872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.61:38417 every 8 connection(s)
I20260809 07:57:53.228892 30778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-1/data/info.pb
I20260809 07:57:53.231925 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 30778
I20260809 07:57:53.232131 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-1/wal/instance
I20260809 07:57:53.232570 29120 external_mini_cluster.cc:1428] Setting key 9cf79cf3cded1cd8b342ae5fa6698e25
I20260809 07:57:53.233839 30873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:53.240000 30873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.112.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.60@KRBTEST.COM" created.
Aug 09 07:57:53 dist-test-slave-vpvm krb5kdc[30641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262273, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.61@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:57:53.263705 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.243085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43474 (local address 127.28.112.62:40725)
0809 07:57:53.243329 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.243342 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.244836 (+ 1494us) server_negotiation.cc:407] Connection header received
0809 07:57:53.246470 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.246473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.246558 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.246689 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.249027 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.250082 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.252555 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.252904 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.253089 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.256965 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:53.256977 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:53.256979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:53.257021 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:53.260394 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.261179 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.261191 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.261195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.261307 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.261787 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.261791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.261793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.262204 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:53.262340 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:53.263152 (+ 812us) server_negotiation.cc:299] Negotiation successful
0809 07:57:53.263400 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 07:57:53.265317 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.243717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:40725 (local address 127.0.0.1:43474)
0809 07:57:53.244512 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.244655 (+ 143us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:53.246112 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:53.246921 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:53.246938 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:53.247597 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:53.248783 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:53.248821 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.250300 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.250304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:53.252399 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:53.252416 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.253165 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.253170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:53.253484 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.254359 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:53.254391 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:53.256729 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:53.260542 (+ 3813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:53.260558 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:53.260574 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:53.261042 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:53.261439 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:53.261444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:53.261446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:53.261662 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:53.262354 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:53.262365 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:53.262772 (+ 407us) client_negotiation.cc:769] Sending connection context
0809 07:57:53.263046 (+ 274us) client_negotiation.cc:240] Negotiation successful
0809 07:57:53.263646 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":161,"threads_started":1}
I20260809 07:57:53.267401 30873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.60.keytab.
Entry for principal kudu/127.28.112.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.60.keytab.
I20260809 07:57:53.269438 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.60:38561
--webserver_interface=127.28.112.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.60.keytab
--principal=kudu/127.28.112.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:40725,127.28.112.61:38417,127.28.112.60:38561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 07:57:53 dist-test-slave-vpvm krb5kdc[30641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262273, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.61@KRBTEST.COM for kudu/127.28.112.61@KRBTEST.COM
I20260809 07:57:53.277323 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.268116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:38417 (local address 127.0.0.1:44512)
0809 07:57:53.268332 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.268345 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:53.268467 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:53.269379 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:53.269393 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:53.269661 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:53.269767 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:53.269773 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.270733 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.270737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:53.271309 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:53.271317 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.272041 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.272043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:53.272122 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.272557 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:53.272573 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:53.273894 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:53.275865 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:53.275867 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:53.275869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:53.276043 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:53.276367 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:53.276370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:53.276385 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:53.276462 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:53.276965 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:53.276967 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:53.277039 (+ 72us) client_negotiation.cc:769] Sending connection context
0809 07:57:53.277110 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 07:57:53.277206 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260809 07:57:53.277518 30881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.268247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44512 (local address 127.28.112.61:38417)
0809 07:57:53.268741 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.268760 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.268795 (+ 35us) server_negotiation.cc:407] Connection header received
0809 07:57:53.268871 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.268876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.269171 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.269343 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.269863 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.270618 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.271415 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.271939 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.272047 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.274069 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:53.274079 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:53.274105 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:53.274132 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:53.275763 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.276148 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.276155 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.276156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.276248 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.276545 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.276548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.276550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.276857 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:53.277015 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:53.277204 (+ 189us) server_negotiation.cc:299] Negotiation successful
0809 07:57:53.277350 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":152,"threads_started":1}
I20260809 07:57:53.279313 30873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } has no permanent_uuid. Determining permanent_uuid...
I20260809 07:57:53.280017 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.279753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:38561 (local address 127.0.0.1:33694)
0809 07:57:53.279869 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.279951 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 07:57:53.280193 30799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.112.60:38561: connect: Connection refused (error 111)
I20260809 07:57:53.280980 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.280767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:38561 (local address 127.0.0.1:33708)
0809 07:57:53.280875 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.280924 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 07:57:53.281127 30873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.60:38561: Network error: Client connection negotiation failed: client connection to 127.28.112.60:38561: connect: Connection refused (error 111)
I20260809 07:57:53.323590 30764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } attempt: 4
Aug 09 07:57:53 dist-test-slave-vpvm krb5kdc[30641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262272, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for kudu/127.28.112.61@KRBTEST.COM
I20260809 07:57:53.330353 30873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } attempt: 1
I20260809 07:57:53.331030 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.330769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:38561 (local address 127.0.0.1:33710)
0809 07:57:53.330900 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.330950 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 07:57:53.331674 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.331472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:38561 (local address 127.0.0.1:33724)
0809 07:57:53.331572 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.331615 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260809 07:57:53.331851 30873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.60:38561: Network error: Client connection negotiation failed: client connection to 127.28.112.60:38561: connect: Connection refused (error 111)
I20260809 07:57:53.333194 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.324193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:38417 (local address 127.0.0.1:44526)
0809 07:57:53.324463 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.324478 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:53.324579 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:53.324849 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:53.324851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:53.325032 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:53.325128 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:53.325134 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.326196 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.326200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:53.326766 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:53.326776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.327376 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.327380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:53.327505 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.328006 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:53.328019 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:53.329463 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:53.331826 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:53.331830 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:53.331832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:53.332068 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:53.332341 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:53.332343 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:53.332344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:53.332425 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:53.332803 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:53.332805 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:53.332886 (+ 81us) client_negotiation.cc:769] Sending connection context
0809 07:57:53.332960 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 07:57:53.333067 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260809 07:57:53.333289 30881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.324295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44526 (local address 127.28.112.61:38417)
0809 07:57:53.324444 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.324449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.324546 (+ 97us) server_negotiation.cc:407] Connection header received
0809 07:57:53.324669 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.324672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.324729 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.324820 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.325239 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.326071 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.326883 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.327250 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.327355 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.329613 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:53.329617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:53.329620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:53.329648 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:53.331720 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.332159 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.332163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.332165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.332256 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.332501 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.332504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.332506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.332724 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:53.332831 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:53.333026 (+ 195us) server_negotiation.cc:299] Negotiation successful
0809 07:57:53.333156 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 07:57:53.334333 30764 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } has no permanent_uuid. Determining permanent_uuid...
I20260809 07:57:53.334975 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.334764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:38561 (local address 127.0.0.1:33728)
0809 07:57:53.334868 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.334915 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260809 07:57:53.335531 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.335345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:38561 (local address 127.0.0.1:33734)
0809 07:57:53.335437 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.335483 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260809 07:57:53.335647 30764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.60:38561: Network error: Client connection negotiation failed: client connection to 127.28.112.60:38561: connect: Connection refused (error 111)
W20260809 07:57:53.383881 30883 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 07:57:53.384155 30883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:53.384191 30883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:53.384253 30883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:53.388396 30883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:53.388461 30883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:53.388489 30883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:53.388511 30883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:53.388540 30883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:53.393816 30883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:40725,127.28.112.61:38417,127.28.112.60:38561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.60.keytab
--principal=kudu/127.28.112.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.60:38561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.112.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:53.394786 30764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } attempt: 1
I20260809 07:57:53.394915 30883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:53.395668 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.395371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:38561 (local address 127.0.0.1:33750)
0809 07:57:53.395511 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.395579 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260809 07:57:53.396032 30883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 07:57:53.396342 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.396151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:38561 (local address 127.0.0.1:33752)
0809 07:57:53.396242 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.396290 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260809 07:57:53.396489 30764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.60:38561: Network error: Client connection negotiation failed: client connection to 127.28.112.60:38561: connect: Connection refused (error 111)
I20260809 07:57:53.400027 30873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } attempt: 2
I20260809 07:57:53.400681 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.400450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:38561 (local address 127.0.0.1:33754)
0809 07:57:53.400557 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.400616 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260809 07:57:53.401351 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.401146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:38561 (local address 127.0.0.1:33766)
0809 07:57:53.401254 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.401295 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 07:57:53.401472 30873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.60:38561: Network error: Client connection negotiation failed: client connection to 127.28.112.60:38561: connect: Connection refused (error 111)
I20260809 07:57:53.404431 30883 server_base.cc:1034] running on GCE node
W20260809 07:57:53.404409 30889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:53.404417 30895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:53.404608 30892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:53.405149 30883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:53.406311 30883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:53.407467 30883 hybrid_clock.cc:648] HybridClock initialized: now 1786262273407449 us; error 30 us; skew 500 ppm
Aug 09 07:57:53 dist-test-slave-vpvm krb5kdc[30641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262273, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:53.410018 30883 init.cc:377] Logged in from keytab as kudu/127.28.112.60@KRBTEST.COM (short username kudu)
I20260809 07:57:53.410857 30883 webserver.cc:460] Webserver started at http://127.28.112.60:44997/ using document root <none> and password file <none>
I20260809 07:57:53.411379 30883 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:53.411445 30883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:53.411659 30883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:53.413854 30883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-2/data/instance:
uuid: "20d5700ef31b414791cb0dc0adeae72c"
format_stamp: "Formatted at 2026-08-09 07:57:53 on dist-test-slave-vpvm"
server_key: "62501cb02553de8a148fdf2f017560b8"
server_key_iv: "8275761e96cb99434275a3e833ed0894"
server_key_version: "encryptionkey@0"
I20260809 07:57:53.414546 30883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-2/wal/instance:
uuid: "20d5700ef31b414791cb0dc0adeae72c"
format_stamp: "Formatted at 2026-08-09 07:57:53 on dist-test-slave-vpvm"
server_key: "62501cb02553de8a148fdf2f017560b8"
server_key_iv: "8275761e96cb99434275a3e833ed0894"
server_key_version: "encryptionkey@0"
I20260809 07:57:53.418548 30883 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 07:57:53.420778 30903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:53.421895 30883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:57:53.422056 30883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-2/wal
uuid: "20d5700ef31b414791cb0dc0adeae72c"
format_stamp: "Formatted at 2026-08-09 07:57:53 on dist-test-slave-vpvm"
server_key: "62501cb02553de8a148fdf2f017560b8"
server_key_iv: "8275761e96cb99434275a3e833ed0894"
server_key_version: "encryptionkey@0"
I20260809 07:57:53.422158 30883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:53.434374 30883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:53.436895 30883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:53.437060 30883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:53.443521 30883 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.60:38561
I20260809 07:57:53.443755 30990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.60:38561 every 8 connection(s)
I20260809 07:57:53.444419 30883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-2/data/info.pb
I20260809 07:57:53.446077 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 30883
I20260809 07:57:53.446201 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786261991120089-29120-0/minicluster-data/master-2/wal/instance
I20260809 07:57:53.446467 29120 external_mini_cluster.cc:1428] Setting key 487a369a0f79f4a03ea5f5052b5f4a92
I20260809 07:57:53.447304 30992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:53.451711 30992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 07:57:53 dist-test-slave-vpvm krb5kdc[30641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.60@KRBTEST.COM
Aug 09 07:57:53 dist-test-slave-vpvm krb5kdc[30641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262273, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.60@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:57:53.460662 30997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.447773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33768 (local address 127.28.112.60:38561)
0809 07:57:53.448175 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.448194 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.448236 (+ 42us) server_negotiation.cc:407] Connection header received
0809 07:57:53.449051 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.449074 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.449462 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.449818 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.450491 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.451948 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.452796 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.453350 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.453490 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.455847 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:53.455852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:53.455860 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:53.455885 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:53.457486 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.458433 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.458439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.458445 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.458573 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.458848 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.458851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.458852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.459187 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:53.459366 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:53.459650 (+ 284us) server_negotiation.cc:299] Negotiation successful
0809 07:57:53.459881 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":112,"threads_started":1}
I20260809 07:57:53.461879 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.453411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43482 (local address 127.28.112.62:40725)
0809 07:57:53.453536 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.453540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.453914 (+ 374us) server_negotiation.cc:407] Connection header received
0809 07:57:53.454113 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.454116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.454159 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.454224 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.454837 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.455339 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.456347 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.456672 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.456769 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.458844 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:53.458849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:53.458852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:53.458879 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:53.460176 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.460623 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.460625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.460626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.460703 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.461082 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.461086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.461087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.461314 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:53.461414 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:53.461673 (+ 259us) server_negotiation.cc:299] Negotiation successful
0809 07:57:53.461770 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 07:57:53.461925 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.453310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:40725 (local address 127.0.0.1:43482)
0809 07:57:53.453792 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.453837 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:53.454013 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:53.454282 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:53.454287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:53.454607 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:53.454727 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:53.454733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.455458 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.455462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:53.456231 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:53.456243 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.456779 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.456783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:53.456881 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.457289 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:53.457300 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:53.458716 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:53.460279 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:53.460286 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:53.460291 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:53.460540 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:53.460901 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:53.460904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:53.460906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:53.460993 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:53.461404 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:53.461409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:53.461493 (+ 84us) client_negotiation.cc:769] Sending connection context
0809 07:57:53.461605 (+ 112us) client_negotiation.cc:240] Negotiation successful
0809 07:57:53.461759 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":91,"threads_started":1}
I20260809 07:57:53.463449 30992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } has no permanent_uuid. Determining permanent_uuid...
I20260809 07:57:53.463622 30764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } attempt: 2
Aug 09 07:57:53 dist-test-slave-vpvm krb5kdc[30641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
Aug 09 07:57:53 dist-test-slave-vpvm krb5kdc[30641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262272, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for kudu/127.28.112.60@KRBTEST.COM
Aug 09 07:57:53 dist-test-slave-vpvm krb5kdc[30641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262273, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.60@KRBTEST.COM for kudu/127.28.112.61@KRBTEST.COM
I20260809 07:57:53.471767 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.462935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43492 (local address 127.28.112.62:40725)
0809 07:57:53.463036 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.463039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.463098 (+ 59us) server_negotiation.cc:407] Connection header received
0809 07:57:53.463197 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.463200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.463241 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.463316 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.463986 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.464608 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.465537 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.465805 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.465896 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.468592 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:53.468596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:53.468598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:53.468624 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:53.470110 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.470548 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.470551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.470553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.470649 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.470957 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.470961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.470963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.471180 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:53.471387 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:53.471531 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 07:57:53.471628 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":41}
I20260809 07:57:53.472550 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.463890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:38417 (local address 127.0.0.1:44536)
0809 07:57:53.464581 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.464593 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:53.464673 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:53.464903 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:53.464906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:53.465062 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:53.465172 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:53.465178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.466109 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.466113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:53.466691 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:53.466701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.467282 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.467286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:53.467385 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.467783 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:53.467794 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:53.469272 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:53.471053 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:53.471056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:53.471058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:53.471242 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:53.471549 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:53.471552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:53.471553 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:53.471648 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:53.472111 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:53.472114 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:53.472202 (+ 88us) client_negotiation.cc:769] Sending connection context
0809 07:57:53.472308 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 07:57:53.472424 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634}
I20260809 07:57:53.472731 30881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.464391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44536 (local address 127.28.112.61:38417)
0809 07:57:53.464506 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.464510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.464665 (+ 155us) server_negotiation.cc:407] Connection header received
0809 07:57:53.464750 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.464753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.464803 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.464878 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.465277 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.465987 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.466807 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.467145 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.467250 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.469412 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:53.469416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:53.469418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:53.469444 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:53.470966 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.471351 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.471355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.471356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.471443 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.471758 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.471761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.471763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.471964 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:53.472035 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:53.472469 (+ 434us) server_negotiation.cc:299] Negotiation successful
0809 07:57:53.472582 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 07:57:53.472762 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.463980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:38561 (local address 127.0.0.1:33770)
0809 07:57:53.464139 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.464153 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:53.464227 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:53.464430 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:53.464433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:53.464576 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:53.464660 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:53.464668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.465452 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.465456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:53.466263 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:53.466273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.466705 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.466708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:53.466780 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.467158 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:53.467171 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:53.468869 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:53.471300 (+ 2431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:53.471303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:53.471305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:53.471531 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:53.471854 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:53.471857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:53.471858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:53.471946 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:53.472353 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:53.472356 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:53.472444 (+ 88us) client_negotiation.cc:769] Sending connection context
0809 07:57:53.472519 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 07:57:53.472621 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":33}
I20260809 07:57:53.473207 30997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.464032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33770 (local address 127.28.112.60:38561)
0809 07:57:53.464119 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.464123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.464192 (+ 69us) server_negotiation.cc:407] Connection header received
0809 07:57:53.464295 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.464297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.464327 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.464391 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.464783 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.465332 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.466361 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.466604 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.466697 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.469011 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:53.469015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:53.469017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:53.469296 (+ 279us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:53.471182 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.471663 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.471666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.471668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.471744 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.472071 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.472073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.472075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.472245 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:53.472907 (+ 662us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:53.472977 (+ 70us) server_negotiation.cc:299] Negotiation successful
0809 07:57:53.473075 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":33}
I20260809 07:57:53.473503 30992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } has no permanent_uuid. Determining permanent_uuid...
I20260809 07:57:53.476826 30764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded: Bootstrap starting.
Aug 09 07:57:53 dist-test-slave-vpvm krb5kdc[30641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262273, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.60@KRBTEST.COM for kudu/127.28.112.60@KRBTEST.COM
Aug 09 07:57:53 dist-test-slave-vpvm krb5kdc[30641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.61@KRBTEST.COM
I20260809 07:57:53.479465 30764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:53.480350 30764 log.cc:824] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:53.481272 30997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.474218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33780 (local address 127.28.112.60:38561)
0809 07:57:53.474298 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.474302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.474314 (+ 12us) server_negotiation.cc:407] Connection header received
0809 07:57:53.474369 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.474372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.474409 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.474479 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.474780 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.475211 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.475864 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.476068 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.476150 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.478185 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:53.478189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:53.478191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:53.478218 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:53.479791 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.480193 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.480196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.480198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.480273 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.480525 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.480528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.480530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.480687 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:53.480770 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:53.481051 (+ 281us) server_negotiation.cc:299] Negotiation successful
0809 07:57:53.481163 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260809 07:57:53.481642 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.473976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:38561 (local address 127.0.0.1:33780)
0809 07:57:53.474217 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.474229 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:53.474296 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:53.474489 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:53.474492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:53.474630 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:53.474687 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:53.474692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.475300 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.475303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:53.475769 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:53.475776 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.476239 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.476242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:53.476309 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.476719 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:53.476733 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:53.478040 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:53.479882 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:53.479885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:53.479887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:53.480114 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:53.480348 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:53.480351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:53.480353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:53.480426 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:53.480801 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:53.480804 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:53.480887 (+ 83us) client_negotiation.cc:769] Sending connection context
0809 07:57:53.481433 (+ 546us) client_negotiation.cc:240] Negotiation successful
0809 07:57:53.481546 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260809 07:57:53.482511 30881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.473189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.28.112.61:38417)
0809 07:57:53.473356 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.473360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.473632 (+ 272us) server_negotiation.cc:407] Connection header received
0809 07:57:53.473724 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.473727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.473773 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.473855 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.474535 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.475154 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.476126 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.476424 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.476758 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.479217 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:53.479220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:53.479223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:53.479246 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:53.480772 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.481152 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.481155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.481157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.481239 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.481542 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.481544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.481546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.481745 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:53.481987 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:53.482264 (+ 277us) server_negotiation.cc:299] Negotiation successful
0809 07:57:53.482364 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55,"thread_start_us":103,"threads_started":1}
I20260809 07:57:53.482879 30764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded: No bootstrap required, opened a new log
I20260809 07:57:53.484443 30992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c: Bootstrap starting.
I20260809 07:57:53.485379 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:57:53.486644 30724 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "15bf553506aa4188b5aee880eec43ded"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43492
I20260809 07:57:53.486820 30764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf553506aa4188b5aee880eec43ded" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } } peers { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } } peers { permanent_uuid: "20d5700ef31b414791cb0dc0adeae72c" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } }
I20260809 07:57:53.487002 30992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:53.487057 30764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15bf553506aa4188b5aee880eec43ded, State: Initialized, Role: FOLLOWER
I20260809 07:57:53.487403 30764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf553506aa4188b5aee880eec43ded" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } } peers { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } } peers { permanent_uuid: "20d5700ef31b414791cb0dc0adeae72c" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } }
I20260809 07:57:53.487826 30992 log.cc:824] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:53.487928 31001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf553506aa4188b5aee880eec43ded" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } } peers { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } } peers { permanent_uuid: "20d5700ef31b414791cb0dc0adeae72c" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } } }
I20260809 07:57:53.488075 31001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:57:53.488735 30764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:53.488809 30724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 07:57:53.488925 30724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:53.489904 30992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c: No bootstrap required, opened a new log
I20260809 07:57:53.490227 30724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf553506aa4188b5aee880eec43ded" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } } peers { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } } peers { permanent_uuid: "20d5700ef31b414791cb0dc0adeae72c" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } }
I20260809 07:57:53.491154 30724 leader_election.cc:288] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [CANDIDATE]: Term 1 election: Requested vote from peers 50ca5d0e7d02407880a4ee86229160b7 (127.28.112.61:38417), 20d5700ef31b414791cb0dc0adeae72c (127.28.112.60:38561)
I20260809 07:57:53.492609 30873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } attempt: 3
I20260809 07:57:53.493762 30992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf553506aa4188b5aee880eec43ded" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } } peers { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } } peers { permanent_uuid: "20d5700ef31b414791cb0dc0adeae72c" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } }
I20260809 07:57:53.494014 30992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d5700ef31b414791cb0dc0adeae72c, State: Initialized, Role: FOLLOWER
I20260809 07:57:53.494351 30992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf553506aa4188b5aee880eec43ded" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } } peers { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } } peers { permanent_uuid: "20d5700ef31b414791cb0dc0adeae72c" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } }
I20260809 07:57:53.495754 30992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c [sys.catalog]: configured and running, proceeding with master startup.
Aug 09 07:57:53 dist-test-slave-vpvm krb5kdc[30641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262273, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.61@KRBTEST.COM for kudu/127.28.112.60@KRBTEST.COM
I20260809 07:57:53.498055 30953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15bf553506aa4188b5aee880eec43ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "20d5700ef31b414791cb0dc0adeae72c"
I20260809 07:57:53.498293 30953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:53.499610 30953 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15bf553506aa4188b5aee880eec43ded in term 1.
W20260809 07:57:53.499812 31016 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:57:53.499895 31016 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:57:53.500833 30686 leader_election.cc:302] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15bf553506aa4188b5aee880eec43ded, 20d5700ef31b414791cb0dc0adeae72c; no voters:
I20260809 07:57:53.495668 30663 master_runner.cc:186] Error getting master registration for 127.28.112.61:38417: OK, instance_id { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" instance_seqno: 1786262273224349 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 07:57:53.502141 30663 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 07:57:53.502718 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.493007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:38561 (local address 127.0.0.1:33786)
0809 07:57:53.493189 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:53.493203 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:53.493288 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:53.493524 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:53.493527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:53.493704 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:53.493804 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:53.493811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.494879 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.494883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:53.495702 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:53.495714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.496303 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.496306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:53.496393 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.496879 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:53.496893 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:53.498376 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:53.500734 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:53.500738 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:53.500740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:53.500983 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:53.501432 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:53.501436 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:53.501439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:53.501531 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:53.502221 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:53.502237 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:53.502337 (+ 100us) client_negotiation.cc:769] Sending connection context
0809 07:57:53.502422 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 07:57:53.502555 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260809 07:57:53.502777 31001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:53.503005 31001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [term 1 LEADER]: Becoming Leader. State: Replica: 15bf553506aa4188b5aee880eec43ded, State: Running, Role: LEADER
I20260809 07:57:53.503357 31001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf553506aa4188b5aee880eec43ded" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } } peers { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } } peers { permanent_uuid: "20d5700ef31b414791cb0dc0adeae72c" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } }
I20260809 07:57:53.503602 30997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.493042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33786 (local address 127.28.112.60:38561)
0809 07:57:53.493150 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.493154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.493247 (+ 93us) server_negotiation.cc:407] Connection header received
0809 07:57:53.493362 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.493364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.493407 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.493915 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.494048 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.494746 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.495853 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.496175 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.496614 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.498522 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:53.498526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:53.498528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:53.498553 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:53.500603 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.501147 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.501154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.501156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.501295 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.501687 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.501690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.501692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.502061 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:53.503020 (+ 959us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:53.503197 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 07:57:53.503361 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 07:57:53.505342 31001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15bf553506aa4188b5aee880eec43ded. Latest consensus state: current_term: 1 leader_uuid: "15bf553506aa4188b5aee880eec43ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf553506aa4188b5aee880eec43ded" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } } peers { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } } peers { permanent_uuid: "20d5700ef31b414791cb0dc0adeae72c" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } } }
I20260809 07:57:53.505482 31001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:53.507102 30873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7: Bootstrap starting.
I20260809 07:57:53.504523 30883 master_runner.cc:186] Error getting master registration for 127.28.112.61:38417: OK, instance_id { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" instance_seqno: 1786262273224349 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 07:57:53.508211 30883 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 07:57:53.509346 31045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:53.509796 30873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:53.510319 31045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:53.510643 30873 log.cc:824] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:53.512897 30873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7: No bootstrap required, opened a new log
I20260809 07:57:53.516623 30873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf553506aa4188b5aee880eec43ded" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } } peers { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } } peers { permanent_uuid: "20d5700ef31b414791cb0dc0adeae72c" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } }
I20260809 07:57:53.516831 30873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ca5d0e7d02407880a4ee86229160b7, State: Initialized, Role: FOLLOWER
I20260809 07:57:53.517148 30873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf553506aa4188b5aee880eec43ded" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } } peers { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } } peers { permanent_uuid: "20d5700ef31b414791cb0dc0adeae72c" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } }
I20260809 07:57:53.518644 30834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15bf553506aa4188b5aee880eec43ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "50ca5d0e7d02407880a4ee86229160b7"
I20260809 07:57:53.518847 30834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7 [term 0 FOLLOWER]: Advancing to term 1
W20260809 07:57:53.520015 31037 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260809 07:57:53.520021 30834 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15bf553506aa4188b5aee880eec43ded in term 1.
I20260809 07:57:53.520583 30873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:53.520625 31024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf553506aa4188b5aee880eec43ded" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } } peers { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } } peers { permanent_uuid: "20d5700ef31b414791cb0dc0adeae72c" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } } }
I20260809 07:57:53.520870 31024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c [sys.catalog]: This master's current role is: FOLLOWER
W20260809 07:57:53.520710 31037 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:57:53.524693 31051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.507586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.28.112.60:38561)
0809 07:57:53.511012 (+ 3426us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.511019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.511362 (+ 343us) server_negotiation.cc:407] Connection header received
0809 07:57:53.511404 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.511407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.511452 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.511528 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.512371 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.513193 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.514266 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.514639 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.514738 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.516431 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:53.516435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:53.516438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:53.516465 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:53.520684 (+ 4219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.522697 (+ 2013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.522701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.522703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.522794 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.523475 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.523479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.523480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.523707 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:53.523826 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:53.524394 (+ 568us) server_negotiation.cc:299] Negotiation successful
0809 07:57:53.524513 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3338,"thread_start_us":99,"threads_started":1}
W20260809 07:57:53.525278 31069 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:57:53.525349 31069 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:57:53.526062 31054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf553506aa4188b5aee880eec43ded" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } } peers { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } } peers { permanent_uuid: "20d5700ef31b414791cb0dc0adeae72c" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } } }
I20260809 07:57:53.526181 31054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:57:53.530922 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.506122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43502 (local address 127.28.112.62:40725)
0809 07:57:53.506235 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.506240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.510276 (+ 4036us) server_negotiation.cc:407] Connection header received
0809 07:57:53.510492 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.510495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.510542 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.510611 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.511830 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.512534 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.515515 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.515860 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.515940 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.521796 (+ 5856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:53.521800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:53.521802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:53.521841 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:53.528168 (+ 6327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.529485 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.529489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.529491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.529587 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.529923 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.529927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.529929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.530178 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:53.530308 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:53.530670 (+ 362us) server_negotiation.cc:299] Negotiation successful
0809 07:57:53.530780 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 07:57:53.532706 30881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:53.506970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44550 (local address 127.28.112.61:38417)
0809 07:57:53.507075 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.507079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.510713 (+ 3634us) server_negotiation.cc:407] Connection header received
0809 07:57:53.510845 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.510848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.510893 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.510957 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.516515 (+ 5558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.517419 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.522175 (+ 4756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:53.522497 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:53.522574 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:53.529178 (+ 6604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:53.529182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:53.529184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:53.529210 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:53.530749 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.531420 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.531424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.531426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.531511 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:53.531814 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:53.531817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:53.531819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:53.532019 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:53.532119 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:53.532467 (+ 348us) server_negotiation.cc:299] Negotiation successful
0809 07:57:53.532572 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 07:57:53.540235 30953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c [term 1 FOLLOWER]: Refusing update from remote peer 15bf553506aa4188b5aee880eec43ded: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:57:53.540926 31040 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [LEADER]: Connected to new peer: Peer: permanent_uuid: "20d5700ef31b414791cb0dc0adeae72c" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:57:53.546517 31024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15bf553506aa4188b5aee880eec43ded. Latest consensus state: current_term: 1 leader_uuid: "15bf553506aa4188b5aee880eec43ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf553506aa4188b5aee880eec43ded" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } } peers { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } } peers { permanent_uuid: "20d5700ef31b414791cb0dc0adeae72c" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } } }
I20260809 07:57:53.546653 31024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:57:53.549714 30834 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7 [term 1 FOLLOWER]: Refusing update from remote peer 15bf553506aa4188b5aee880eec43ded: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:57:53.550257 31040 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [LEADER]: Connected to new peer: Peer: permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:57:53.554065 31054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15bf553506aa4188b5aee880eec43ded. Latest consensus state: current_term: 1 leader_uuid: "15bf553506aa4188b5aee880eec43ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf553506aa4188b5aee880eec43ded" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } } peers { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } } peers { permanent_uuid: "20d5700ef31b414791cb0dc0adeae72c" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } } }
I20260809 07:57:53.554176 31054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:57:53.561622 31040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15bf553506aa4188b5aee880eec43ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf553506aa4188b5aee880eec43ded" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } } peers { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } } peers { permanent_uuid: "20d5700ef31b414791cb0dc0adeae72c" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } } }
I20260809 07:57:53.561755 31040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:53.562845 31054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15bf553506aa4188b5aee880eec43ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf553506aa4188b5aee880eec43ded" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } } peers { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } } peers { permanent_uuid: "20d5700ef31b414791cb0dc0adeae72c" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } } }
I20260809 07:57:53.562951 31054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:57:53.563472 31056 mvcc.cc:205] Tried to move back new op lower bound from 7316530272412381184 to 7316530272274034688. Current Snapshot: MvccSnapshot[applied={T|T < 7316530272412381184}]
I20260809 07:57:53.564446 31075 mvcc.cc:205] Tried to move back new op lower bound from 7316530272412381184 to 7316530272274034688. Current Snapshot: MvccSnapshot[applied={T|T < 7316530272412381184}]
I20260809 07:57:53.566393 31001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15bf553506aa4188b5aee880eec43ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf553506aa4188b5aee880eec43ded" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } } peers { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } } peers { permanent_uuid: "20d5700ef31b414791cb0dc0adeae72c" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } } }
I20260809 07:57:53.566504 31001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:53.567108 31045 catalog_manager.cc:1347] Generated new cluster ID: 083296f8cbe44a33a8ff633684306a23
I20260809 07:57:53.567171 31045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:53.568483 31024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15bf553506aa4188b5aee880eec43ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bf553506aa4188b5aee880eec43ded" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 40725 } } peers { permanent_uuid: "50ca5d0e7d02407880a4ee86229160b7" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 38417 } } peers { permanent_uuid: "20d5700ef31b414791cb0dc0adeae72c" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 38561 } } }
I20260809 07:57:53.568596 31024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:57:53.570230 31073 mvcc.cc:205] Tried to move back new op lower bound from 7316530272412381184 to 7316530272274034688. Current Snapshot: MvccSnapshot[applied={T|T < 7316530272412381184}]
I20260809 07:57:53.608754 31045 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:53.610073 31045 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:53.625600 31045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15bf553506aa4188b5aee880eec43ded: Generated new TSK 0
I20260809 07:57:53.626569 31045 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 07:57:53.656247 30775 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:53.653613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43510 (local address 127.28.112.62:40725)
0809 07:57:53.653739 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.653744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.654417 (+ 673us) server_negotiation.cc:407] Connection header received
0809 07:57:53.654637 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.654641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.654691 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.654743 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.656113 (+ 1370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43510: BlockingRecv error: recv got EOF from 127.0.0.1:43510 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260809 07:57:53.658648 30881 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:53.656905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.28.112.61:38417)
0809 07:57:53.657025 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.657029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.657700 (+ 671us) server_negotiation.cc:407] Connection header received
0809 07:57:53.657865 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.657869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.657915 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.657985 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.658521 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44566: BlockingRecv error: recv got EOF from 127.0.0.1:44566 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260809 07:57:53.660190 31051 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:53.657432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.28.112.60:38561)
0809 07:57:53.657543 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.657546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.659259 (+ 1713us) server_negotiation.cc:407] Connection header received
0809 07:57:53.659502 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.659505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.659557 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.659606 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.660076 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33806: BlockingRecv error: recv got EOF from 127.0.0.1:33806 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260809 07:57:53.667847 30881 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:53.666339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.28.112.61:38417)
0809 07:57:53.666463 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.666467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.666907 (+ 440us) server_negotiation.cc:407] Connection header received
0809 07:57:53.667126 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.667130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.667202 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.667252 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.667748 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44570: BlockingRecv error: recv got EOF from 127.0.0.1:44570 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260809 07:57:53.673000 30881 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:53.671476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44580 (local address 127.28.112.61:38417)
0809 07:57:53.671575 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.671579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.672051 (+ 472us) server_negotiation.cc:407] Connection header received
0809 07:57:53.672234 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.672237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.672285 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.672333 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.672906 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44580: BlockingRecv error: recv got EOF from 127.0.0.1:44580 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260809 07:57:53.678674 30881 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:53.677181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44588 (local address 127.28.112.61:38417)
0809 07:57:53.677274 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.677278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.677751 (+ 473us) server_negotiation.cc:407] Connection header received
0809 07:57:53.677928 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.677931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.678027 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.678080 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.678587 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44588: BlockingRecv error: recv got EOF from 127.0.0.1:44588 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260809 07:57:53.686259 30881 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:53.684785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44598 (local address 127.28.112.61:38417)
0809 07:57:53.684907 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.684911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.685364 (+ 453us) server_negotiation.cc:407] Connection header received
0809 07:57:53.685554 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.685557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.685603 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.685650 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.686169 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44598: BlockingRecv error: recv got EOF from 127.0.0.1:44598 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260809 07:57:53.698545 30881 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:53.696674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.28.112.61:38417)
0809 07:57:53.696785 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.696790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.697393 (+ 603us) server_negotiation.cc:407] Connection header received
0809 07:57:53.697637 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.697641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.697697 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.697755 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.698438 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44604: BlockingRecv error: recv got EOF from 127.0.0.1:44604 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260809 07:57:53.717864 30881 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:53.716758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.28.112.61:38417)
0809 07:57:53.716885 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.716889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.717073 (+ 184us) server_negotiation.cc:407] Connection header received
0809 07:57:53.717199 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.717201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.717246 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.717312 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.717755 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44616: BlockingRecv error: recv got EOF from 127.0.0.1:44616 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260809 07:57:53.752492 30881 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:53.751534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44632 (local address 127.28.112.61:38417)
0809 07:57:53.751701 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.751705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.751778 (+ 73us) server_negotiation.cc:407] Connection header received
0809 07:57:53.751892 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.751895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.751941 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.752009 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.752375 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44632: BlockingRecv error: recv got EOF from 127.0.0.1:44632 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260809 07:57:53.819603 30881 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:53.818571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44642 (local address 127.28.112.61:38417)
0809 07:57:53.818764 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.818768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.818843 (+ 75us) server_negotiation.cc:407] Connection header received
0809 07:57:53.818969 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.818972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.819018 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.819090 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.819480 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44642: BlockingRecv error: recv got EOF from 127.0.0.1:44642 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260809 07:57:53.950712 30881 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:53.949635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44644 (local address 127.28.112.61:38417)
0809 07:57:53.949806 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:53.949810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:53.949913 (+ 103us) server_negotiation.cc:407] Connection header received
0809 07:57:53.950076 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:53.950079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:53.950128 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:53.950198 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:53.950574 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44644: BlockingRecv error: recv got EOF from 127.0.0.1:44644 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260809 07:57:54.209616 30881 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:54.208636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44654 (local address 127.28.112.61:38417)
0809 07:57:54.208750 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:54.208754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:54.208870 (+ 116us) server_negotiation.cc:407] Connection header received
0809 07:57:54.208988 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:54.208990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:54.209037 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:54.209108 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:54.209489 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44654: BlockingRecv error: recv got EOF from 127.0.0.1:44654 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 07:57:54.522461 31037 catalog_manager.cc:1259] Loaded cluster ID: 083296f8cbe44a33a8ff633684306a23
I20260809 07:57:54.522554 31037 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c: loading cluster ID for follower catalog manager: success
I20260809 07:57:54.524307 31037 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c: acquiring CA information for follower catalog manager: success
I20260809 07:57:54.525547 31037 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 20d5700ef31b414791cb0dc0adeae72c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 07:57:54.526427 31069 catalog_manager.cc:1259] Loaded cluster ID: 083296f8cbe44a33a8ff633684306a23
I20260809 07:57:54.526484 31069 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7: loading cluster ID for follower catalog manager: success
I20260809 07:57:54.527890 31069 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7: acquiring CA information for follower catalog manager: success
I20260809 07:57:54.528837 31069 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 50ca5d0e7d02407880a4ee86229160b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 07:57:54.727468 31181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:54.723642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44668 (local address 127.28.112.61:38417)
0809 07:57:54.723883 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:54.723887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:54.723952 (+ 65us) server_negotiation.cc:407] Connection header received
0809 07:57:54.724108 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:54.724111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:54.724245 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:54.724348 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:57:54.724645 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:54.725167 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:54.726076 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:54.726302 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:54.726376 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:54.726621 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:57:54.727103 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:57:54.727198 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:54.727266 (+ 68us) server_negotiation.cc:299] Negotiation successful
0809 07:57:54.727314 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":78,"threads_started":1}
I20260809 07:57:54.733160 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:54.730202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33818 (local address 127.28.112.60:38561)
0809 07:57:54.730438 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:54.730441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:54.730454 (+ 13us) server_negotiation.cc:407] Connection header received
0809 07:57:54.730530 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:54.730532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:54.730644 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:54.730732 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:57:54.730875 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:54.731352 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:54.732066 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:54.732266 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:54.732331 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:54.732528 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:57:54.732876 (+ 348us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:57:54.732954 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:54.732982 (+ 28us) server_negotiation.cc:299] Negotiation successful
0809 07:57:54.733023 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":77,"threads_started":1}
I20260809 07:57:54.739143 31181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:54.735729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44678 (local address 127.28.112.61:38417)
0809 07:57:54.735846 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:54.735850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:54.736059 (+ 209us) server_negotiation.cc:407] Connection header received
0809 07:57:54.736252 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:54.736255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:54.736301 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:54.736379 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:57:54.736592 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:54.737164 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:54.738040 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:54.738377 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:54.738476 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:54.738704 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:57:54.738766 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:57:54.738829 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:54.738972 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 07:57:54.739024 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 07:57:54.739835 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:54.736176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33830 (local address 127.28.112.60:38561)
0809 07:57:54.736271 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:54.736275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:54.736416 (+ 141us) server_negotiation.cc:407] Connection header received
0809 07:57:54.736568 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:54.736571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:54.736615 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:54.736698 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:57:54.736914 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:54.737624 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:54.738820 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:54.739109 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:54.739196 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:54.739443 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:57:54.739513 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:57:54.739582 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:54.739678 (+ 96us) server_negotiation.cc:299] Negotiation successful
0809 07:57:54.739726 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 07:57:54.740249 31200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:54.735724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43520 (local address 127.28.112.62:40725)
0809 07:57:54.735939 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:54.735944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:54.735958 (+ 14us) server_negotiation.cc:407] Connection header received
0809 07:57:54.736175 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:54.736178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:54.736322 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:54.736442 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:57:54.736799 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:54.737621 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:54.738588 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:54.738910 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:54.739002 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:54.739470 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:57:54.739752 (+ 282us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:57:54.739830 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:54.740091 (+ 261us) server_negotiation.cc:299] Negotiation successful
0809 07:57:54.740144 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":58,"threads_started":1}
I20260809 07:57:54.742884 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 30663
I20260809 07:57:54.754032 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 30778
I20260809 07:57:54.765887 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 30883
2026-08-09T07:57:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3380 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 07:57:54 dist-test-slave-vpvm krb5kdc[31207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 07:57:54 dist-test-slave-vpvm krb5kdc[31207](info): set up 2 sockets
Aug 09 07:57:54 dist-test-slave-vpvm krb5kdc[31207](info): commencing operation
krb5kdc: starting...
W20260809 07:57:55.897787 29120 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/test-user.keytab.
Aug 09 07:57:55 dist-test-slave-vpvm krb5kdc[31207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T07:57:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T07:57:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.112.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
Entry for principal kudu/127.28.112.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab.
I20260809 07:57:56.012164 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:39089
--webserver_interface=127.28.112.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:39089,127.28.112.61:46603,127.28.112.60:34049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 07:57:56.119725 31228 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 07:57:56.119990 31228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:56.120026 31228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:56.120087 31228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:56.124001 31228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:56.124050 31228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:56.124078 31228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:56.124099 31228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:56.124126 31228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:56.129088 31228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:39089,127.28.112.61:46603,127.28.112.60:34049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.62.keytab
--principal=kudu/127.28.112.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.62:39089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.112.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:56.130038 31228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:56.130992 31228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:56.138926 31234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:56.139075 31228 server_base.cc:1034] running on GCE node
W20260809 07:57:56.138932 31237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:56.138954 31240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:56.139778 31228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:56.140749 31228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:56.141892 31228 hybrid_clock.cc:648] HybridClock initialized: now 1786262276141875 us; error 42 us; skew 500 ppm
Aug 09 07:57:56 dist-test-slave-vpvm krb5kdc[31207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:56.146911 31228 init.cc:377] Logged in from keytab as kudu/127.28.112.62@KRBTEST.COM (short username kudu)
I20260809 07:57:56.147730 31228 webserver.cc:460] Webserver started at http://127.28.112.62:34111/ using document root <none> and password file <none>
I20260809 07:57:56.148228 31228 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:56.148290 31228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:56.148474 31228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:56.150454 31228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-0/data/instance:
uuid: "78e72e7ff8f64dcc8ea083c9052f6759"
format_stamp: "Formatted at 2026-08-09 07:57:56 on dist-test-slave-vpvm"
server_key: "1fd19774dd3f15b6fde9ec3bf80111eb"
server_key_iv: "f457135c0ffe79f1c7d1f7a907c2ca8c"
server_key_version: "encryptionkey@0"
I20260809 07:57:56.150990 31228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-0/wal/instance:
uuid: "78e72e7ff8f64dcc8ea083c9052f6759"
format_stamp: "Formatted at 2026-08-09 07:57:56 on dist-test-slave-vpvm"
server_key: "1fd19774dd3f15b6fde9ec3bf80111eb"
server_key_iv: "f457135c0ffe79f1c7d1f7a907c2ca8c"
server_key_version: "encryptionkey@0"
I20260809 07:57:56.154428 31228 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 07:57:56.156651 31248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:56.157774 31228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:57:56.157881 31228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-0/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-0/wal
uuid: "78e72e7ff8f64dcc8ea083c9052f6759"
format_stamp: "Formatted at 2026-08-09 07:57:56 on dist-test-slave-vpvm"
server_key: "1fd19774dd3f15b6fde9ec3bf80111eb"
server_key_iv: "f457135c0ffe79f1c7d1f7a907c2ca8c"
server_key_version: "encryptionkey@0"
I20260809 07:57:56.158018 31228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:56.174902 31228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:56.177522 31228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:56.177701 31228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:56.184401 31228 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.62:39089
I20260809 07:57:56.184662 31327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.62:39089 every 8 connection(s)
I20260809 07:57:56.185315 31228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-0/data/info.pb
I20260809 07:57:56.187564 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 31228
I20260809 07:57:56.187691 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-0/wal/instance
I20260809 07:57:56.187958 29120 external_mini_cluster.cc:1428] Setting key 35fbbd5ef7153f9cd7c3c611d22b3bc1
I20260809 07:57:56.188066 31331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:56.192137 31331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.112.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.61@KRBTEST.COM" created.
Aug 09 07:57:56 dist-test-slave-vpvm krb5kdc[31207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
I20260809 07:57:56.206166 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.194790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.28.112.62:39089)
0809 07:57:56.195130 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.195141 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.195182 (+ 41us) server_negotiation.cc:407] Connection header received
0809 07:57:56.195832 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.195836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.196016 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.196173 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.196983 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.197867 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.198915 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.199258 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.199400 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.201742 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:56.201750 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:56.201763 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:56.201790 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:56.203316 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.203742 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.203747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.203748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.203843 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.204117 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.204120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.204122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.204450 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:56.204566 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:56.204931 (+ 365us) server_negotiation.cc:299] Negotiation successful
0809 07:57:56.205176 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":252,"thread_start_us":153,"threads_started":1}
I20260809 07:57:56.206166 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.194219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:39089 (local address 127.0.0.1:34504)
0809 07:57:56.194661 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.194711 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:56.195554 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:56.196189 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:56.196195 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:56.196592 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:56.196815 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:56.196827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.198006 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.198009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:56.198802 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:56.198812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.199393 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.199397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:56.199502 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.200040 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:56.200062 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:56.201572 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:56.203416 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:56.203424 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:56.203434 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:56.203645 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:56.203927 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:56.203930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:56.203932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:56.204029 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:56.204556 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:56.204562 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:56.204690 (+ 128us) client_negotiation.cc:769] Sending connection context
0809 07:57:56.204857 (+ 167us) client_negotiation.cc:240] Negotiation successful
0809 07:57:56.205232 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":116,"thread_start_us":99,"threads_started":1}
I20260809 07:57:56.208290 31331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } has no permanent_uuid. Determining permanent_uuid...
I20260809 07:57:56.208935 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.208703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:46603 (local address 127.0.0.1:45942)
0809 07:57:56.208810 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.208876 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:46603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 07:57:56.209112 31253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.112.61:46603: connect: Connection refused (error 111)
I20260809 07:57:56.209889 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.209688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:46603 (local address 127.0.0.1:45944)
0809 07:57:56.209794 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.209834 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:46603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260809 07:57:56.210029 31331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.61:46603: Network error: Client connection negotiation failed: client connection to 127.28.112.61:46603: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.61.keytab.
Entry for principal kudu/127.28.112.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.61.keytab.
I20260809 07:57:56.212561 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.61:46603
--webserver_interface=127.28.112.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.61.keytab
--principal=kudu/127.28.112.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:39089,127.28.112.61:46603,127.28.112.60:34049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 07:57:56.259280 31331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } attempt: 1
I20260809 07:57:56.260205 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.259876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:46603 (local address 127.0.0.1:45960)
0809 07:57:56.260031 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.260101 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:46603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260809 07:57:56.261168 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.260872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:46603 (local address 127.0.0.1:45974)
0809 07:57:56.260992 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.261091 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:46603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260809 07:57:56.261349 31331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.61:46603: Network error: Client connection negotiation failed: client connection to 127.28.112.61:46603: connect: Connection refused (error 111)
I20260809 07:57:56.329519 31331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } attempt: 2
I20260809 07:57:56.330399 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.330124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:46603 (local address 127.0.0.1:45984)
0809 07:57:56.330263 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.330323 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:46603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260809 07:57:56.331040 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.330859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:46603 (local address 127.0.0.1:45994)
0809 07:57:56.330945 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.330986 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.61:46603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260809 07:57:56.331179 31331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.61:46603: Network error: Client connection negotiation failed: client connection to 127.28.112.61:46603: connect: Connection refused (error 111)
W20260809 07:57:56.332975 31340 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 07:57:56.333207 31340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:56.333233 31340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:56.333273 31340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:56.337162 31340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:56.337214 31340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:56.337232 31340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:56.337247 31340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:56.337260 31340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:56.342660 31340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:39089,127.28.112.61:46603,127.28.112.60:34049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.61.keytab
--principal=kudu/127.28.112.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.61:46603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.112.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:56.343608 31340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:56.344529 31340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 07:57:56.352402 31351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:56.352471 31349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:56.352402 31353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:56.352725 31340 server_base.cc:1034] running on GCE node
I20260809 07:57:56.353229 31340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:56.354316 31340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:56.355471 31340 hybrid_clock.cc:648] HybridClock initialized: now 1786262276355453 us; error 32 us; skew 500 ppm
Aug 09 07:57:56 dist-test-slave-vpvm krb5kdc[31207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:56.360746 31340 init.cc:377] Logged in from keytab as kudu/127.28.112.61@KRBTEST.COM (short username kudu)
I20260809 07:57:56.361621 31340 webserver.cc:460] Webserver started at http://127.28.112.61:40379/ using document root <none> and password file <none>
I20260809 07:57:56.362212 31340 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:56.362274 31340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:56.362466 31340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:56.364384 31340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-1/data/instance:
uuid: "d97306a1f31242b1b16e51449adc5ce1"
format_stamp: "Formatted at 2026-08-09 07:57:56 on dist-test-slave-vpvm"
server_key: "274b2e010ea875ba7b3701d517f8561e"
server_key_iv: "c712b573a30e3623ecffac9d118438f4"
server_key_version: "encryptionkey@0"
I20260809 07:57:56.364892 31340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-1/wal/instance:
uuid: "d97306a1f31242b1b16e51449adc5ce1"
format_stamp: "Formatted at 2026-08-09 07:57:56 on dist-test-slave-vpvm"
server_key: "274b2e010ea875ba7b3701d517f8561e"
server_key_iv: "c712b573a30e3623ecffac9d118438f4"
server_key_version: "encryptionkey@0"
I20260809 07:57:56.368394 31340 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 07:57:56.370714 31364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:56.371989 31340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:57:56.372112 31340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-1/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-1/wal
uuid: "d97306a1f31242b1b16e51449adc5ce1"
format_stamp: "Formatted at 2026-08-09 07:57:56 on dist-test-slave-vpvm"
server_key: "274b2e010ea875ba7b3701d517f8561e"
server_key_iv: "c712b573a30e3623ecffac9d118438f4"
server_key_version: "encryptionkey@0"
I20260809 07:57:56.372200 31340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:56.398420 31340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:56.400980 31340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:56.401154 31340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:56.407639 31340 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.61:46603
I20260809 07:57:56.407889 31454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.61:46603 every 8 connection(s)
I20260809 07:57:56.408524 31340 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-1/data/info.pb
I20260809 07:57:56.408758 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 31340
I20260809 07:57:56.408865 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-1/wal/instance
I20260809 07:57:56.409179 29120 external_mini_cluster.cc:1428] Setting key 0d61042b24825f90511d2bff3dd27c34
I20260809 07:57:56.411242 31455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:56.415799 31455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.112.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.112.60@KRBTEST.COM" created.
I20260809 07:57:56.422338 31331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } attempt: 3
Aug 09 07:57:56 dist-test-slave-vpvm krb5kdc[31207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.61@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
Aug 09 07:57:56 dist-test-slave-vpvm krb5kdc[31207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for kudu/127.28.112.61@KRBTEST.COM
I20260809 07:57:56.430939 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.417933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34508 (local address 127.28.112.62:39089)
0809 07:57:56.418081 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.418085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.418904 (+ 819us) server_negotiation.cc:407] Connection header received
0809 07:57:56.419831 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.419834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.419883 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.419948 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.421031 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.421543 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.423160 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.423499 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.423585 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.426524 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:56.426527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:56.426529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:56.426552 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:56.428042 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.428654 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.428659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.428660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.428744 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.429100 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.429103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.429105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.429933 (+ 828us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:56.430066 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:56.430648 (+ 582us) server_negotiation.cc:299] Negotiation successful
0809 07:57:56.430769 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 07:57:56.431898 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.418297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:39089 (local address 127.0.0.1:34508)
0809 07:57:56.418750 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.418801 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:56.419628 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:56.420085 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:56.420091 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:56.420417 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:56.420874 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:56.420888 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.421670 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.421674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:56.423009 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:56.423019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.423797 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.423802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:56.424020 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.424671 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:56.424691 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:56.426390 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:56.428176 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:56.428183 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:56.428197 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:56.428525 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:56.428835 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:56.428838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:56.428839 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:56.429004 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:56.430063 (+ 1059us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:56.430070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:56.430359 (+ 289us) client_negotiation.cc:769] Sending connection context
0809 07:57:56.430564 (+ 205us) client_negotiation.cc:240] Negotiation successful
0809 07:57:56.430882 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":110,"threads_started":1}
I20260809 07:57:56.433110 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.422785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:46603 (local address 127.0.0.1:46000)
0809 07:57:56.422952 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.422965 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:56.423042 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:56.424321 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:56.424325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:56.424507 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:56.424632 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:56.424652 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.426138 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.426142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:56.426938 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:56.426950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.427702 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.427706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:56.427817 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.428268 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:56.428280 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:56.429578 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:56.431586 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:56.431589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:56.431591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:56.431825 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:56.432131 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:56.432135 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:56.432136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:56.432215 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:56.432664 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:56.432668 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:56.432764 (+ 96us) client_negotiation.cc:769] Sending connection context
0809 07:57:56.432852 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 07:57:56.432946 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260809 07:57:56.433243 31464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.423360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46000 (local address 127.28.112.61:46603)
0809 07:57:56.423855 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.423866 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.423887 (+ 21us) server_negotiation.cc:407] Connection header received
0809 07:57:56.423927 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.423932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.424122 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.424796 (+ 674us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.424902 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.425939 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.427068 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.427575 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.427685 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.429738 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:56.429751 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:56.429765 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:56.429792 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:56.431484 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.431943 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.431947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.431949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.432046 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.432299 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.432303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.432305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.432553 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:56.432685 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:56.432944 (+ 259us) server_negotiation.cc:299] Negotiation successful
0809 07:57:56.433077 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":149,"threads_started":1}
I20260809 07:57:56.433231 31455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } has no permanent_uuid. Determining permanent_uuid...
I20260809 07:57:56.435060 31331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } has no permanent_uuid. Determining permanent_uuid...
I20260809 07:57:56.435676 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.435455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:34049 (local address 127.0.0.1:37650)
0809 07:57:56.435555 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.435605 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260809 07:57:56.436311 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.436100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:34049 (local address 127.0.0.1:37660)
0809 07:57:56.436218 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.436263 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260809 07:57:56.436417 31331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.60:34049: Network error: Client connection negotiation failed: client connection to 127.28.112.60:34049: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.112.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.60.keytab.
Entry for principal kudu/127.28.112.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.60.keytab.
Aug 09 07:57:56 dist-test-slave-vpvm krb5kdc[31207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.61@KRBTEST.COM for kudu/127.28.112.61@KRBTEST.COM
I20260809 07:57:56.438426 29120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4dUq1L/build/debug/bin/kudu
/tmp/dist-test-task4dUq1L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.60:34049
--webserver_interface=127.28.112.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.60.keytab
--principal=kudu/127.28.112.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.112.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.112.62:39089,127.28.112.61:46603,127.28.112.60:34049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 07:57:56.441180 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.433699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:46603 (local address 127.0.0.1:46010)
0809 07:57:56.433870 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.433882 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:56.433942 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:56.434311 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:56.434314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:56.434488 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:56.434609 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:56.434616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.435411 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.435414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:56.435997 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:56.436007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.436509 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.436512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:56.436598 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.436912 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:56.436921 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:56.438171 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:56.439841 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:56.439844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:56.439846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:56.440075 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:56.440313 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:56.440316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:56.440318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:56.440404 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:56.440687 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:56.440690 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:56.440778 (+ 88us) client_negotiation.cc:769] Sending connection context
0809 07:57:56.440883 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 07:57:56.441008 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260809 07:57:56.441187 31464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.434028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46010 (local address 127.28.112.61:46603)
0809 07:57:56.434103 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.434107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.434115 (+ 8us) server_negotiation.cc:407] Connection header received
0809 07:57:56.434149 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.434152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.434184 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.434276 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.434719 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.435317 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.436101 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.436408 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.436498 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.438294 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:56.438298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:56.438301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:56.438326 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:56.439716 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.440171 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.440174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.440175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.440242 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.440473 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.440475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.440476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.440611 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:56.440681 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:56.440924 (+ 243us) server_negotiation.cc:299] Negotiation successful
0809 07:57:56.441059 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":24}
I20260809 07:57:56.441897 31455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } has no permanent_uuid. Determining permanent_uuid...
I20260809 07:57:56.442490 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.442281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:34049 (local address 127.0.0.1:37670)
0809 07:57:56.442375 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.442443 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260809 07:57:56.442667 31367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.112.60:34049: connect: Connection refused (error 111)
I20260809 07:57:56.443456 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.443276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:34049 (local address 127.0.0.1:37686)
0809 07:57:56.443368 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.443411 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260809 07:57:56.443599 31455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.60:34049: Network error: Client connection negotiation failed: client connection to 127.28.112.60:34049: connect: Connection refused (error 111)
I20260809 07:57:56.467545 31331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } attempt: 1
I20260809 07:57:56.468189 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.467944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:34049 (local address 127.0.0.1:37690)
0809 07:57:56.468067 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.468120 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260809 07:57:56.468869 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.468700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:34049 (local address 127.0.0.1:37706)
0809 07:57:56.468782 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.468822 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260809 07:57:56.468977 31331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.60:34049: Network error: Client connection negotiation failed: client connection to 127.28.112.60:34049: connect: Connection refused (error 111)
I20260809 07:57:56.492822 31455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } attempt: 1
I20260809 07:57:56.493686 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.493402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:34049 (local address 127.0.0.1:37712)
0809 07:57:56.493545 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.493606 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260809 07:57:56.494354 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.494147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:34049 (local address 127.0.0.1:37718)
0809 07:57:56.494247 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.494295 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260809 07:57:56.494474 31455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.60:34049: Network error: Client connection negotiation failed: client connection to 127.28.112.60:34049: connect: Connection refused (error 111)
I20260809 07:57:56.544131 31331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } attempt: 2
I20260809 07:57:56.544946 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.544658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:34049 (local address 127.0.0.1:37732)
0809 07:57:56.544811 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.544870 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 07:57:56.545707 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.545476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:34049 (local address 127.0.0.1:37740)
0809 07:57:56.545602 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.545651 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260809 07:57:56.545833 31331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.60:34049: Network error: Client connection negotiation failed: client connection to 127.28.112.60:34049: connect: Connection refused (error 111)
W20260809 07:57:56.550201 31465 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 07:57:56.550441 31465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 07:57:56.550478 31465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 07:57:56.550546 31465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 07:57:56.554404 31465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 07:57:56.554453 31465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 07:57:56.554487 31465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 07:57:56.554517 31465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 07:57:56.554539 31465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 07:57:56.559597 31465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.112.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.112.62:39089,127.28.112.61:46603,127.28.112.60:34049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/krb5kdc/kudu_127.28.112.60.keytab
--principal=kudu/127.28.112.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.112.60:34049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.112.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 07:01:58 UTC on ccycloud.quasar-fnohdt.root.comops.site
build id 4575851
I20260809 07:57:56.560545 31465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 07:57:56.561507 31465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 07:57:56.562611 31455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } attempt: 2
I20260809 07:57:56.563314 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.563064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:34049 (local address 127.0.0.1:37756)
0809 07:57:56.563195 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.563246 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 07:57:56.563973 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.563778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:34049 (local address 127.0.0.1:37768)
0809 07:57:56.563874 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.563917 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.112.60:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260809 07:57:56.564152 31455 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.112.60:34049: Network error: Client connection negotiation failed: client connection to 127.28.112.60:34049: connect: Connection refused (error 111)
W20260809 07:57:56.570142 31476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 07:57:56.570132 31475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:56.570319 31465 server_base.cc:1034] running on GCE node
W20260809 07:57:56.570137 31478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 07:57:56.570899 31465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 07:57:56.571929 31465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 07:57:56.573098 31465 hybrid_clock.cc:648] HybridClock initialized: now 1786262276573087 us; error 36 us; skew 500 ppm
Aug 09 07:57:56 dist-test-slave-vpvm krb5kdc[31207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 07:57:56.575654 31465 init.cc:377] Logged in from keytab as kudu/127.28.112.60@KRBTEST.COM (short username kudu)
I20260809 07:57:56.576573 31465 webserver.cc:460] Webserver started at http://127.28.112.60:42921/ using document root <none> and password file <none>
I20260809 07:57:56.577108 31465 fs_manager.cc:359] Metadata directory not provided
I20260809 07:57:56.577173 31465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 07:57:56.577363 31465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 07:57:56.579334 31465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-2/data/instance:
uuid: "239a3595843a4f6dba88ca20d41ec746"
format_stamp: "Formatted at 2026-08-09 07:57:56 on dist-test-slave-vpvm"
server_key: "e378f03f5b3e4440a595c2da880b5620"
server_key_iv: "2d6d8aaa78ccb91a2a526f69e2abc57b"
server_key_version: "encryptionkey@0"
I20260809 07:57:56.579847 31465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-2/wal/instance:
uuid: "239a3595843a4f6dba88ca20d41ec746"
format_stamp: "Formatted at 2026-08-09 07:57:56 on dist-test-slave-vpvm"
server_key: "e378f03f5b3e4440a595c2da880b5620"
server_key_iv: "2d6d8aaa78ccb91a2a526f69e2abc57b"
server_key_version: "encryptionkey@0"
I20260809 07:57:56.583330 31465 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 07:57:56.585493 31488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 07:57:56.586638 31465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 07:57:56.586750 31465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-2/data,/tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-2/wal
uuid: "239a3595843a4f6dba88ca20d41ec746"
format_stamp: "Formatted at 2026-08-09 07:57:56 on dist-test-slave-vpvm"
server_key: "e378f03f5b3e4440a595c2da880b5620"
server_key_iv: "2d6d8aaa78ccb91a2a526f69e2abc57b"
server_key_version: "encryptionkey@0"
I20260809 07:57:56.586836 31465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 07:57:56.617921 31465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 07:57:56.620540 31465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 07:57:56.620715 31465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 07:57:56.627665 31465 rpc_server.cc:307] RPC server started. Bound to: 127.28.112.60:34049
I20260809 07:57:56.627904 31565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.112.60:34049 every 8 connection(s)
I20260809 07:57:56.628618 31465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-2/data/info.pb
I20260809 07:57:56.631309 31566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 07:57:56.635066 29120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4dUq1L/build/debug/bin/kudu as pid 31465
I20260809 07:57:56.635187 29120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4dUq1L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786261991120089-29120-0/minicluster-data/master-2/wal/instance
I20260809 07:57:56.634915 31566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } has no permanent_uuid. Determining permanent_uuid...
I20260809 07:57:56.635516 29120 external_mini_cluster.cc:1428] Setting key c952da1571146e6a8fbfe8f0a2217c0a
Aug 09 07:57:56 dist-test-slave-vpvm krb5kdc[31207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.60@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
Aug 09 07:57:56 dist-test-slave-vpvm krb5kdc[31207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.60@KRBTEST.COM
I20260809 07:57:56.644971 31331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } attempt: 3
I20260809 07:57:56.647887 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.636849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34510 (local address 127.28.112.62:39089)
0809 07:57:56.636984 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.636989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.637439 (+ 450us) server_negotiation.cc:407] Connection header received
0809 07:57:56.638156 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.638159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.638214 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.638288 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.639179 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.639672 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.640727 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.641053 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.641145 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.643853 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:56.643857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:56.643859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:56.643882 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:56.645539 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.646199 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.646203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.646205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.646300 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.646699 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.646703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.646705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.646927 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:56.647021 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:56.647585 (+ 564us) server_negotiation.cc:299] Negotiation successful
0809 07:57:56.647708 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 07:57:56.648954 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.636825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37784 (local address 127.28.112.60:34049)
0809 07:57:56.637232 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.637247 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.637280 (+ 33us) server_negotiation.cc:407] Connection header received
0809 07:57:56.637918 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.637934 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.638170 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.638333 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.639041 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.639973 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.640963 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.641401 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.641503 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.644318 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:56.644323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:56.644332 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:56.644353 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:56.646151 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.646628 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.646635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.646636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.646733 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.647038 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.647041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.647043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.647340 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:56.647530 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:56.647742 (+ 212us) server_negotiation.cc:299] Negotiation successful
0809 07:57:56.648151 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":114,"threads_started":1}
I20260809 07:57:56.648958 31572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.636826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.62:39089 (local address 127.0.0.1:34510)
0809 07:57:56.637308 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.637352 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:56.637926 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:56.638364 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:56.638369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:56.638782 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:56.639036 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:56.639048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.639771 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.639775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:56.640566 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:56.640575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.641188 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.641192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:56.641363 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.641934 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:56.641969 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:56.643695 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:56.645674 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:56.645684 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:56.645691 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:56.646047 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:56.646381 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:56.646384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:56.646386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:56.646554 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:56.647029 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:56.647035 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:56.647298 (+ 263us) client_negotiation.cc:769] Sending connection context
0809 07:57:56.648091 (+ 793us) client_negotiation.cc:240] Negotiation successful
0809 07:57:56.648269 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"spinlock_wait_cycles":23424,"thread_start_us":150,"threads_started":1}
Aug 09 07:57:56 dist-test-slave-vpvm krb5kdc[31207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.62@KRBTEST.COM for kudu/127.28.112.60@KRBTEST.COM
I20260809 07:57:56.651032 31566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } has no permanent_uuid. Determining permanent_uuid...
I20260809 07:57:56.654745 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.645353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:34049 (local address 127.0.0.1:37790)
0809 07:57:56.645530 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.645543 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:56.645617 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:56.646064 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:56.646067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:56.646240 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:56.646349 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:56.646358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.647239 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.647243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:56.647966 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:56.647980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.648586 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.648590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:56.648684 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.649389 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:56.649400 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:56.650706 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:56.653230 (+ 2524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:56.653234 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:56.653236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:56.653459 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:56.653759 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:56.653763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:56.653765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:56.653864 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:56.654296 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:56.654299 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:56.654385 (+ 86us) client_negotiation.cc:769] Sending connection context
0809 07:57:56.654470 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 07:57:56.654596 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260809 07:57:56.655289 31455 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } attempt: 3
I20260809 07:57:56.655359 31573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.645438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.28.112.60:34049)
0809 07:57:56.645828 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.645836 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.645849 (+ 13us) server_negotiation.cc:407] Connection header received
0809 07:57:56.645880 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.645883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.645925 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.646037 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.646465 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.647130 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.648111 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.648456 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.648567 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.650861 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:56.650866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:56.650868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:56.650907 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:56.653122 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.653571 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.653574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.653576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.653670 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.653980 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.653983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.653985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.654189 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:56.655005 (+ 816us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:56.655081 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 07:57:56.655198 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":103,"threads_started":1}
Aug 09 07:57:56 dist-test-slave-vpvm krb5kdc[31207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.62@KRBTEST.COM
Aug 09 07:57:56 dist-test-slave-vpvm krb5kdc[31207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.60@KRBTEST.COM for kudu/127.28.112.61@KRBTEST.COM
I20260809 07:57:56.659041 31331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759: Bootstrap starting.
I20260809 07:57:56.660606 31572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.651513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.61:46603 (local address 127.0.0.1:46018)
0809 07:57:56.651656 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.651668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:56.651736 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:56.652269 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:56.652272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:56.652434 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:56.652528 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:56.652534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.653458 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.653461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:56.654150 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:56.654160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.654628 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.654632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:56.654737 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.655927 (+ 1190us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:56.655942 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:56.657126 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:56.659057 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:56.659060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:56.659062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:56.659332 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:56.659673 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:56.659675 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:56.659677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:56.659765 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:56.660143 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:56.660146 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:56.660235 (+ 89us) client_negotiation.cc:769] Sending connection context
0809 07:57:56.660348 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 07:57:56.660469 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260809 07:57:56.660701 31464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.651888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46018 (local address 127.28.112.61:46603)
0809 07:57:56.652046 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.652050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.652064 (+ 14us) server_negotiation.cc:407] Connection header received
0809 07:57:56.652106 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.652109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.652167 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.652238 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.652639 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.653349 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.654263 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.654512 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.654619 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.657267 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:56.657271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:56.657274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:56.657300 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:56.658938 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.659457 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.659461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.659462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.659550 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.659852 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.659856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.659858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.660044 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:56.660132 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:56.660431 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 07:57:56.660539 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Aug 09 07:57:56 dist-test-slave-vpvm krb5kdc[31207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.61@KRBTEST.COM for kudu/127.28.112.60@KRBTEST.COM
I20260809 07:57:56.661756 31566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } has no permanent_uuid. Determining permanent_uuid...
I20260809 07:57:56.661932 31331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:56.662853 31331 log.cc:824] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:56.665237 31573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.655715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.28.112.60:34049)
0809 07:57:56.655811 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.655816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.656250 (+ 434us) server_negotiation.cc:407] Connection header received
0809 07:57:56.656286 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.656289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.656328 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.656405 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.656779 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.657324 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.658924 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.659274 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.659375 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.661837 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:56.661841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:56.661844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:56.661871 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:56.663552 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.664081 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.664085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.664086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.664178 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.664470 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.664474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.664476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.664641 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:56.664731 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:56.665003 (+ 272us) server_negotiation.cc:299] Negotiation successful
0809 07:57:56.665102 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 07:57:56.665237 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.655672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:34049 (local address 127.0.0.1:37796)
0809 07:57:56.656067 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.656083 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:56.656165 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:56.656430 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:56.656434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:56.656597 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:56.656669 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:56.656675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.657464 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.657468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:56.658741 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:56.658755 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.659383 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.659387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:56.659479 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.660177 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:56.660187 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:56.661670 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:56.663688 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:56.663692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:56.663694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:56.663945 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:56.664269 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:56.664273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:56.664275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:56.664378 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:56.664747 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:56.664751 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:56.664839 (+ 88us) client_negotiation.cc:769] Sending connection context
0809 07:57:56.664942 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 07:57:56.665070 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320}
I20260809 07:57:56.666416 31331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759: No bootstrap required, opened a new log
I20260809 07:57:56.666877 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.651014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.28.112.62:39089)
0809 07:57:56.651108 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.651111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.651271 (+ 160us) server_negotiation.cc:407] Connection header received
0809 07:57:56.651302 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.651305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.651344 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.651409 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.652149 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.652809 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.653819 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.654086 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.654179 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.663111 (+ 8932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:56.663115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:56.663117 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:56.663155 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:56.664693 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.665130 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.665133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.665134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.665222 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.665522 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.665525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.665527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.665741 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:56.665890 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:56.666617 (+ 727us) server_negotiation.cc:299] Negotiation successful
0809 07:57:56.666741 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 07:57:56.668342 31455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1: Bootstrap starting.
Aug 09 07:57:56 dist-test-slave-vpvm krb5kdc[31207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.112.60@KRBTEST.COM for kudu/127.28.112.60@KRBTEST.COM
I20260809 07:57:56.670650 31331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } } peers { permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } } peers { permanent_uuid: "239a3595843a4f6dba88ca20d41ec746" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } }
I20260809 07:57:56.670881 31331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e72e7ff8f64dcc8ea083c9052f6759, State: Initialized, Role: FOLLOWER
I20260809 07:57:56.671131 31455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:56.671224 31331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } } peers { permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } } peers { permanent_uuid: "239a3595843a4f6dba88ca20d41ec746" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } }
I20260809 07:57:56.672003 31455 log.cc:824] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:56.672633 31331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [sys.catalog]: configured and running, proceeding with master startup.
Aug 09 07:57:56 dist-test-slave-vpvm krb5kdc[31207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786262275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.112.61@KRBTEST.COM
I20260809 07:57:56.674010 31572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.662233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.112.60:34049 (local address 127.0.0.1:37812)
0809 07:57:56.662459 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 07:57:56.662471 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 07:57:56.662540 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 07:57:56.663529 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 07:57:56.663532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 07:57:56.663697 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0809 07:57:56.663779 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:56.663786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.665728 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.665732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:56.666534 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 07:57:56.666546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.667428 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.667432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 07:57:56.667507 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.667948 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 07:57:56.667964 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 07:57:56.669420 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 07:57:56.672245 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:56.672248 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:56.672250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:56.672501 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:56.672874 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 07:57:56.672877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 07:57:56.672879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 07:57:56.672985 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 07:57:56.673425 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 07:57:56.673428 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 07:57:56.673514 (+ 86us) client_negotiation.cc:769] Sending connection context
0809 07:57:56.673691 (+ 177us) client_negotiation.cc:240] Negotiation successful
0809 07:57:56.673820 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260809 07:57:56.674234 31455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1: No bootstrap required, opened a new log
I20260809 07:57:56.675350 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.662347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.28.112.60:34049)
0809 07:57:56.663324 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.663328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.663341 (+ 13us) server_negotiation.cc:407] Connection header received
0809 07:57:56.663373 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.663376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.663413 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.663478 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.663896 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.664616 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.666663 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.667211 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.667322 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.669571 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:56.669575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:56.669578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:56.669601 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:56.672056 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.672617 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.672620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.672622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.672764 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.673097 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.673100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.673102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.673293 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:56.673388 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:56.674374 (+ 986us) server_negotiation.cc:299] Negotiation successful
0809 07:57:56.675209 (+ 835us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909}
W20260809 07:57:56.678610 31595 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:57:56.678685 31595 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:57:56.679121 31464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.668030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46028 (local address 127.28.112.61:46603)
0809 07:57:56.668120 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.668124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.668371 (+ 247us) server_negotiation.cc:407] Connection header received
0809 07:57:56.668407 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.668410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.668446 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.668505 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.669284 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.670016 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.671015 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.671312 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.671406 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.674159 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:56.674163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:56.674165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:56.674190 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:56.675767 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.676779 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.676782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.676783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.676867 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.677172 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.677176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.677178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.677398 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:56.678555 (+ 1157us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:56.678846 (+ 291us) server_negotiation.cc:299] Negotiation successful
0809 07:57:56.678973 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260809 07:57:56.679131 31257 tablet.cc:2369] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 07:57:56.679011 31576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } } peers { permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } } peers { permanent_uuid: "239a3595843a4f6dba88ca20d41ec746" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } } }
I20260809 07:57:56.679656 31576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:57:56.681329 31566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746: Bootstrap starting.
I20260809 07:57:56.683779 31566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746: Neither blocks nor log segments found. Creating new log.
I20260809 07:57:56.684448 31566 log.cc:824] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746: Log is configured to *not* fsync() on all Append() calls
I20260809 07:57:56.682224 31228 master_runner.cc:186] Error getting master registration for 127.28.112.61:46603: OK, instance_id { permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" instance_seqno: 1786262276405881 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 07:57:56.685426 31228 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 07:57:56.685288 31455 raft_consensus.cc:348] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } } peers { permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } } peers { permanent_uuid: "239a3595843a4f6dba88ca20d41ec746" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } }
I20260809 07:57:56.685498 31455 raft_consensus.cc:729] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97306a1f31242b1b16e51449adc5ce1, State: Initialized, Role: FOLLOWER
I20260809 07:57:56.685817 31455 consensus_queue.cc:260] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } } peers { permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } } peers { permanent_uuid: "239a3595843a4f6dba88ca20d41ec746" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } }
I20260809 07:57:56.686367 31605 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } } peers { permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } } peers { permanent_uuid: "239a3595843a4f6dba88ca20d41ec746" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } } }
I20260809 07:57:56.686707 31605 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:57:56.687188 31455 sys_catalog.cc:564] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:56.688195 29120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 07:57:56.689122 31566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746: No bootstrap required, opened a new log
I20260809 07:57:56.689378 31288 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "78e72e7ff8f64dcc8ea083c9052f6759"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34524
I20260809 07:57:56.689607 31288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 07:57:56.689723 31288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:56.690960 31288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } } peers { permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } } peers { permanent_uuid: "239a3595843a4f6dba88ca20d41ec746" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } }
I20260809 07:57:56.691781 31288 leader_election.cc:288] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [CANDIDATE]: Term 1 election: Requested vote from peers d97306a1f31242b1b16e51449adc5ce1 (127.28.112.61:46603), 239a3595843a4f6dba88ca20d41ec746 (127.28.112.60:34049)
I20260809 07:57:56.692232 31412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d97306a1f31242b1b16e51449adc5ce1"
I20260809 07:57:56.692466 31412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:56.693835 31412 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e72e7ff8f64dcc8ea083c9052f6759 in term 1.
I20260809 07:57:56.694347 31253 leader_election.cc:302] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78e72e7ff8f64dcc8ea083c9052f6759, d97306a1f31242b1b16e51449adc5ce1; no voters:
I20260809 07:57:56.694694 31576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 07:57:56.694968 31576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [term 1 LEADER]: Becoming Leader. State: Replica: 78e72e7ff8f64dcc8ea083c9052f6759, State: Running, Role: LEADER
I20260809 07:57:56.695596 31576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } } peers { permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } } peers { permanent_uuid: "239a3595843a4f6dba88ca20d41ec746" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } }
I20260809 07:57:56.695596 31566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } } peers { permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } } peers { permanent_uuid: "239a3595843a4f6dba88ca20d41ec746" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } }
I20260809 07:57:56.695919 31566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 239a3595843a4f6dba88ca20d41ec746, State: Initialized, Role: FOLLOWER
I20260809 07:57:56.696275 31566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } } peers { permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } } peers { permanent_uuid: "239a3595843a4f6dba88ca20d41ec746" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } }
I20260809 07:57:56.697533 31566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 07:57:56.697584 31525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "239a3595843a4f6dba88ca20d41ec746"
I20260809 07:57:56.697814 31525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746 [term 0 FOLLOWER]: Advancing to term 1
I20260809 07:57:56.699170 31525 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e72e7ff8f64dcc8ea083c9052f6759 in term 1.
I20260809 07:57:56.701045 31576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78e72e7ff8f64dcc8ea083c9052f6759. Latest consensus state: current_term: 1 leader_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } } peers { permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } } peers { permanent_uuid: "239a3595843a4f6dba88ca20d41ec746" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } } }
I20260809 07:57:56.701161 31576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:56.710206 31654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 07:57:56.711306 31654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 07:57:56.712191 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.696208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34536 (local address 127.28.112.62:39089)
0809 07:57:56.696307 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.696311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.702370 (+ 6059us) server_negotiation.cc:407] Connection header received
0809 07:57:56.702858 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.702861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.702908 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.702959 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.705087 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.705718 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.706574 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.706885 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.706996 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.708644 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:56.708648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:56.708651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:56.708677 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:56.710262 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.710762 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.710766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.710768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.710857 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.711193 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.711196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.711198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.711412 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:56.711525 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:56.711925 (+ 400us) server_negotiation.cc:299] Negotiation successful
0809 07:57:56.712039 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 07:57:56.714311 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.694791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.28.112.60:34049)
0809 07:57:56.694942 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.694948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.702586 (+ 7638us) server_negotiation.cc:407] Connection header received
0809 07:57:56.703879 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.703882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.703931 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.704114 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.706601 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.707234 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.708082 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.708405 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.708493 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.710587 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:56.710591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:56.710594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:56.710620 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:56.712452 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.712968 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.712971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.712973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.713043 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.713336 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.713339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.713341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.713556 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:56.713682 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:56.714038 (+ 356us) server_negotiation.cc:299] Negotiation successful
0809 07:57:56.714159 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 07:57:56.714555 31464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:56.696033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.28.112.61:46603)
0809 07:57:56.696134 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.696137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.699377 (+ 3240us) server_negotiation.cc:407] Connection header received
0809 07:57:56.699689 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.699693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.699740 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.699786 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.700781 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.701468 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.708684 (+ 7216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:56.708991 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:56.709066 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:56.710639 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 07:57:56.710643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 07:57:56.710645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 07:57:56.710667 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 07:57:56.712153 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.712925 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.712928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.712930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.713016 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 07:57:56.713606 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 07:57:56.713610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 07:57:56.713611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 07:57:56.713815 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 07:57:56.713917 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:56.714302 (+ 385us) server_negotiation.cc:299] Negotiation successful
0809 07:57:56.714407 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 07:57:56.717001 31412 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1 [term 1 FOLLOWER]: Refusing update from remote peer 78e72e7ff8f64dcc8ea083c9052f6759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:57:56.717054 31525 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746 [term 1 FOLLOWER]: Refusing update from remote peer 78e72e7ff8f64dcc8ea083c9052f6759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 07:57:56.717176 31630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } } peers { permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } } peers { permanent_uuid: "239a3595843a4f6dba88ca20d41ec746" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } } }
I20260809 07:57:56.717280 31630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 07:57:56.717020 31628 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:57:56.717411 31628 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:57:56.717707 31627 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 07:57:56.717962 31576 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "239a3595843a4f6dba88ca20d41ec746" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 07:57:56.720198 31652 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 07:57:56.720271 31652 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 07:57:56.721560 31605 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78e72e7ff8f64dcc8ea083c9052f6759. Latest consensus state: current_term: 1 leader_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } } peers { permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } } peers { permanent_uuid: "239a3595843a4f6dba88ca20d41ec746" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } } }
I20260809 07:57:56.721657 31605 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:57:56.722038 31630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78e72e7ff8f64dcc8ea083c9052f6759. Latest consensus state: current_term: 1 leader_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } } peers { permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } } peers { permanent_uuid: "239a3595843a4f6dba88ca20d41ec746" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } } }
I20260809 07:57:56.722136 31630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:57:56.724438 31576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } } peers { permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } } peers { permanent_uuid: "239a3595843a4f6dba88ca20d41ec746" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } } }
I20260809 07:57:56.724543 31576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:56.725395 31630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } } peers { permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } } peers { permanent_uuid: "239a3595843a4f6dba88ca20d41ec746" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } } }
I20260809 07:57:56.725492 31630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:57:56.727278 31654 catalog_manager.cc:1347] Generated new cluster ID: 7ff45191f068478c9cc6a5a534bcdc77
I20260809 07:57:56.727349 31654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 07:57:56.727958 31658 mvcc.cc:205] Tried to move back new op lower bound from 7316530285425119232 to 7316530285349969920. Current Snapshot: MvccSnapshot[applied={T|T < 7316530285425119232}]
I20260809 07:57:56.728644 31664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } } peers { permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } } peers { permanent_uuid: "239a3595843a4f6dba88ca20d41ec746" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } } }
I20260809 07:57:56.728765 31664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759 [sys.catalog]: This master's current role is: LEADER
I20260809 07:57:56.730815 31605 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e72e7ff8f64dcc8ea083c9052f6759" member_type: VOTER last_known_addr { host: "127.28.112.62" port: 39089 } } peers { permanent_uuid: "d97306a1f31242b1b16e51449adc5ce1" member_type: VOTER last_known_addr { host: "127.28.112.61" port: 46603 } } peers { permanent_uuid: "239a3595843a4f6dba88ca20d41ec746" member_type: VOTER last_known_addr { host: "127.28.112.60" port: 34049 } } }
I20260809 07:57:56.730926 31605 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 07:57:56.731678 31660 mvcc.cc:205] Tried to move back new op lower bound from 7316530285425119232 to 7316530285349969920. Current Snapshot: MvccSnapshot[applied={T|T < 7316530285425119232}]
I20260809 07:57:56.737617 31654 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 07:57:56.738875 31654 catalog_manager.cc:1504] Loading token signing keys...
I20260809 07:57:56.751886 31654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78e72e7ff8f64dcc8ea083c9052f6759: Generated new TSK 0
I20260809 07:57:56.752676 31654 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 07:57:56.778470 31336 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:56.775345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34552 (local address 127.28.112.62:39089)
0809 07:57:56.775467 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.775471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.777426 (+ 1955us) server_negotiation.cc:407] Connection header received
0809 07:57:56.777612 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.777616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.777664 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.777714 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.778339 (+ 625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34552: BlockingRecv error: recv got EOF from 127.0.0.1:34552 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260809 07:57:56.779801 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:56.776119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.28.112.61:46603)
0809 07:57:56.776246 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.776251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.778946 (+ 2695us) server_negotiation.cc:407] Connection header received
0809 07:57:56.779114 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.779117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.779163 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.779210 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.779690 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46038: BlockingRecv error: recv got EOF from 127.0.0.1:46038 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260809 07:57:56.781068 31571 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:56.776802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.28.112.60:34049)
0809 07:57:56.776906 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.776920 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.780250 (+ 3330us) server_negotiation.cc:407] Connection header received
0809 07:57:56.780402 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.780405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.780452 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.780500 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.780959 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37824: BlockingRecv error: recv got EOF from 127.0.0.1:37824 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260809 07:57:56.785334 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:56.783897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46050 (local address 127.28.112.61:46603)
0809 07:57:56.784001 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.784004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.784469 (+ 465us) server_negotiation.cc:407] Connection header received
0809 07:57:56.784672 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.784675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.784745 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.784792 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.785250 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46050: BlockingRecv error: recv got EOF from 127.0.0.1:46050 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260809 07:57:56.791572 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:56.790031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46056 (local address 127.28.112.61:46603)
0809 07:57:56.790119 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.790122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.790722 (+ 600us) server_negotiation.cc:407] Connection header received
0809 07:57:56.790884 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.790887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.790943 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.790988 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.791491 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46056: BlockingRecv error: recv got EOF from 127.0.0.1:46056 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260809 07:57:56.796964 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:56.795583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.28.112.61:46603)
0809 07:57:56.795672 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.795676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.796172 (+ 496us) server_negotiation.cc:407] Connection header received
0809 07:57:56.796334 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.796338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.796381 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.796436 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.796871 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46064: BlockingRecv error: recv got EOF from 127.0.0.1:46064 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260809 07:57:56.804136 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:56.802730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46076 (local address 127.28.112.61:46603)
0809 07:57:56.802849 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.802853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.803360 (+ 507us) server_negotiation.cc:407] Connection header received
0809 07:57:56.803525 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.803528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.803571 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.803615 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.804056 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46076: BlockingRecv error: recv got EOF from 127.0.0.1:46076 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260809 07:57:56.815392 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:56.813867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46084 (local address 127.28.112.61:46603)
0809 07:57:56.814055 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.814059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.814544 (+ 485us) server_negotiation.cc:407] Connection header received
0809 07:57:56.814707 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.814710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.814753 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.814800 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.815316 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46084: BlockingRecv error: recv got EOF from 127.0.0.1:46084 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260809 07:57:56.835842 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:56.833834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46088 (local address 127.28.112.61:46603)
0809 07:57:56.834012 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.834018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.834715 (+ 697us) server_negotiation.cc:407] Connection header received
0809 07:57:56.834964 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.834969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.835030 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.835089 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.835728 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46088: BlockingRecv error: recv got EOF from 127.0.0.1:46088 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260809 07:57:56.872627 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:56.870624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.28.112.61:46603)
0809 07:57:56.870763 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.870768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.871486 (+ 718us) server_negotiation.cc:407] Connection header received
0809 07:57:56.871719 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.871723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.871781 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.871839 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.872500 (+ 661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46092: BlockingRecv error: recv got EOF from 127.0.0.1:46092 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260809 07:57:56.940214 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:56.939170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46102 (local address 127.28.112.61:46603)
0809 07:57:56.939315 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:56.939320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:56.939467 (+ 147us) server_negotiation.cc:407] Connection header received
0809 07:57:56.939600 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:56.939602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:56.939652 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:56.939722 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:56.940093 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46102: BlockingRecv error: recv got EOF from 127.0.0.1:46102 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260809 07:57:57.071455 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:57.070381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46104 (local address 127.28.112.61:46603)
0809 07:57:57.070522 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:57.070527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:57.070665 (+ 138us) server_negotiation.cc:407] Connection header received
0809 07:57:57.070789 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:57.070792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:57.070843 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:57.070911 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:57.071333 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46104: BlockingRecv error: recv got EOF from 127.0.0.1:46104 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260809 07:57:57.330677 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 07:57:57.329665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46108 (local address 127.28.112.61:46603)
0809 07:57:57.329801 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:57.329804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:57.329878 (+ 74us) server_negotiation.cc:407] Connection header received
0809 07:57:57.330025 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:57.330027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:57.330077 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:57.330159 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 07:57:57.330545 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46108: BlockingRecv error: recv got EOF from 127.0.0.1:46108 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 07:57:57.718756 31628 catalog_manager.cc:1259] Loaded cluster ID: 7ff45191f068478c9cc6a5a534bcdc77
I20260809 07:57:57.718842 31628 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1: loading cluster ID for follower catalog manager: success
I20260809 07:57:57.720690 31628 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1: acquiring CA information for follower catalog manager: success
I20260809 07:57:57.721496 31652 catalog_manager.cc:1259] Loaded cluster ID: 7ff45191f068478c9cc6a5a534bcdc77
I20260809 07:57:57.721568 31652 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746: loading cluster ID for follower catalog manager: success
I20260809 07:57:57.722226 31628 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d97306a1f31242b1b16e51449adc5ce1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 07:57:57.723223 31652 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746: acquiring CA information for follower catalog manager: success
I20260809 07:57:57.724164 31652 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 239a3595843a4f6dba88ca20d41ec746: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 07:57:57.848347 31775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:57.844905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46116 (local address 127.28.112.61:46603)
0809 07:57:57.845125 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:57.845129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:57.845162 (+ 33us) server_negotiation.cc:407] Connection header received
0809 07:57:57.845299 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:57.845302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:57.845433 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:57.845534 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:57:57.845743 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:57.846299 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:57.847069 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:57.847279 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:57.847348 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:57.847589 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:57:57.848040 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:57:57.848130 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:57.848172 (+ 42us) server_negotiation.cc:299] Negotiation successful
0809 07:57:57.848220 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":64,"threads_started":1}
I20260809 07:57:57.854537 31783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:57.851188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37830 (local address 127.28.112.60:34049)
0809 07:57:57.851434 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:57.851438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:57.851469 (+ 31us) server_negotiation.cc:407] Connection header received
0809 07:57:57.851549 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:57.851552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:57.851683 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:57.851782 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:57:57.851983 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:57.852468 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:57.853242 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:57.853449 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:57.853533 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:57.853790 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:57:57.854184 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:57:57.854286 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:57.854350 (+ 64us) server_negotiation.cc:299] Negotiation successful
0809 07:57:57.854400 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":81,"threads_started":1}
I20260809 07:57:57.861186 31783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:57.857575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37844 (local address 127.28.112.60:34049)
0809 07:57:57.857688 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:57.857692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:57.857707 (+ 15us) server_negotiation.cc:407] Connection header received
0809 07:57:57.857815 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:57.857818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:57.857865 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:57.857927 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:57:57.858238 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:57.858750 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:57.860029 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:57.860290 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:57.860378 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:57.860689 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:57:57.860762 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:57:57.860808 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:57.860992 (+ 184us) server_negotiation.cc:299] Negotiation successful
0809 07:57:57.861042 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 07:57:57.861670 31795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:57.857218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34566 (local address 127.28.112.62:39089)
0809 07:57:57.857507 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:57.857511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:57.857840 (+ 329us) server_negotiation.cc:407] Connection header received
0809 07:57:57.858057 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:57.858061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:57.858202 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:57.858290 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:57:57.858670 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:57.859256 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:57.860122 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:57.860476 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:57.860559 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:57.860873 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:57:57.861195 (+ 322us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:57:57.861261 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:57.861470 (+ 209us) server_negotiation.cc:299] Negotiation successful
0809 07:57:57.861534 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":71,"threads_started":1}
I20260809 07:57:57.861711 31775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 07:57:57.857238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46124 (local address 127.28.112.61:46603)
0809 07:57:57.857345 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0809 07:57:57.857349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 07:57:57.857571 (+ 222us) server_negotiation.cc:407] Connection header received
0809 07:57:57.857781 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 07:57:57.857784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 07:57:57.857840 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 07:57:57.857909 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 07:57:57.858133 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:57.858771 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:57.860047 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 07:57:57.860351 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 07:57:57.860444 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 07:57:57.860703 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 07:57:57.860768 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 07:57:57.860808 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0809 07:57:57.861561 (+ 753us) server_negotiation.cc:299] Negotiation successful
0809 07:57:57.861614 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 07:57:57.864755 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 31228
I20260809 07:57:57.878788 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 31340
I20260809 07:57:57.890583 29120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4dUq1L/build/debug/bin/kudu with pid 31465
2026-08-09T07:57:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3121 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6501 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (286768 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST